Páginas

quinta-feira, setembro 13, 2018

quarta-feira, agosto 22, 2018

Jupyter + Pyspark: link

Precisando colocar o Pyspark pra trabalhar no Jupyter encontrei este link: https://blog.sicara.com/get-started-pyspark-jupyter-guide-tutorial-ae2fe84f594f

Muito bom e muito direto.

Diretamente ao ponto:

import findspark
findspark.init()

sábado, julho 28, 2018

Python: condense dir file contents into one output file

Dica rápida: juntar conteúdo de todos os arquivos de um diretório em um unico arquivo de output:


import os


def condense_files(path, output_file_name=None):
    """Condense all file contents inside a path into one output file"""
    default_file_name = "CONDENSED"
    output_file_name = output_file_name or default_file_name
    file_list = os.listdir(path)
    if os.path.isabs(output_file_name):
        output_file_path = output_file_name
    else:
        output_file_path = os.path.join(path, output_file_name)
    if os.path.isfile(output_file_path):
        raise FileExistsError("Alread exists %s" % output_file_path)

    if os.path.isdir(output_file_path):
        output_file_path = os.path.join(output_file_path, default_file_name)

    with open(output_file_path, "w") as output_file:
        for file_name in file_list:
            with open(os.path.join(path, file_name), "r") as in_file:
                output_file.write(in_file.read())
    return output_file_path


if __name__ == '__main__':
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('--input')
    parser.add_argument('--output')
    args = parser.parse_args()
    condense_files(args.input, args.output)

terça-feira, julho 24, 2018

Random string generation Python and Django

Pure Python exemple:


import string
import random


def random_string(lenth=10):
    return ''.join(random.choices(string.ascii_uppercase + string.digits, k=10))

Ref.: https://stackoverflow.com/questions/2257441/random-string-generation-with-upper-case-letters-and-digits-in-python


If you are using Django, it ready to use in crypto module:


from django.utils.crypto import get_random_string


The raw code:


def get_random_string(length=12,
                      allowed_chars='abcdefghijklmnopqrstuvwxyz'
                                    'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'):
    """
    Return a securely generated random string.

    The default length of 12 with the a-z, A-Z, 0-9 character set returns
    a 71-bit value. log_2((26+26+10)^12) =~ 71 bits
    """
    if not using_sysrandom:
        # This is ugly, and a hack, but it makes things better than
        # the alternative of predictability. This re-seeds the PRNG
        # using a value that is hard for an attacker to predict, every
        # time a random string is required. This may change the
        # properties of the chosen random sequence slightly, but this
        # is better than absolute predictability.
        random.seed(
            hashlib.sha256(
                ('%s%s%s' % (random.getstate(), time.time(), settings.SECRET_KEY)).encode()
            ).digest()
        )
    return ''.join(random.choice(allowed_chars) for i in range(length))




Permissões customizadas no Django (2018)

Há alguns anos escrevi aqui um post sobre criação de permissões do Django com South. Hoje em dia as migrações estão integradas ao Django e as formas possíveis para criação de permissões são mais simples e mais elegantes.

Como documentado AQUI basta adicionar as permissões desejadas ao class Meta do model. Depois disso um make migrations e por fim migrate e pronto.

DJANGO CUSTOM PERMISSIONS 2018

Some years ago i wrote HERE about Django migrations using south. Since Django 1.4, migrations were integrated to it, and we have another way. You can check this out HERE in Django's documentation. Steps below!

1 - Models class Meta:
from django.db import models
from django.conf import settings


class UserProfile(models.Model):
    user = models.OneToOneField(settings.AUTH_USER_MODEL, editable=False, on_delete=models.PROTECT)

    class Meta:
        permissions = (("can_read_own_contents", "Can read own contents"),
                       ("can_read_others_contents", "Can read others contents"),)






2 - Make migrations:
$ python manage.py makemigrations 
Migrations for 'user_profile': 
 user_profile/migrations/0001_initial.py 
   - Create model UserProfile

3 - Check migration data:


from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):
    initial = True

    dependencies = [
        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
    ]

    operations = [
        migrations.CreateModel(
            name='UserProfile',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('user', models.OneToOneField(editable=False, on_delete=django.db.models.deletion.PROTECT,
                                              to=settings.AUTH_USER_MODEL)),
            ],
            options={
                'permissions': (('can_read_own_contents', 'Can read own contents'),
                                ('can_read_others_contents', 'Can read others contents')),
            },
        ),
    ]






4 - Migrate:


Done!






sexta-feira, julho 13, 2018

Configurando PySpark no PyCharm

Dica rápida de configuração do Apache Spark no Pycharm:

1 - Abra as configurações do projeto (File -> Settings)




2 - Em "Project Structure" clique em "+ Add Content Root"

3 - Navegue até o diretório de instalação do Spark no computador e clique em "python" e depois "ok"


4 - Navegue novamente até o diretório de instalação do Spark dentro de python/lib selecione o py4j e "ok" novamente




Deve funcionar!

quarta-feira, julho 04, 2018

Django Rest Framework ViewSets - Mais Fácil

Há cerca de um ano venho trabalhando diretamente com APIs em Django. As ViewSets do rest sempre me pareceram confusas, não entendia bem o modelo, mas me parece que as coisas ficaram mais fáceis agora, vou tentar explicar como usar uma ModeViewSet:

Passo 1 - O model:
Um model padrão do Django com um campo nome

from django.db import models
from messaging.managers import MessageManager


class MessageGroups(models.Model):
    name = models.CharField(max_length=30, unique=True)

    def __str__(self):
        return self.name

Passo 2: A View (ViewSet):
Um ViewSet com queryset padrão trazendo todos os grupos cadastrados

from rest_framework import viewsets

from messaging.models import MessageGroups
from messaging.serializers import MessageGroupSerializer


class MessageGroupViewSet(viewsets.ModelViewSet):
    serializer_class = MessageGroupSerializer
    queryset = MessageGroups.objects.all()

Passo 3: O Urls.py:
No URLs.py do app ou do projeto deve ser adicionada a chamada para DefaultRouter e registrar a url do ViewSet (e foi aqui que me embananei). Depois disso, o urlpatterns deve ser acrescido do router.urls. É muito simples, mas eu relamente me compliquei nessa parte

from django.urls import path, include
from rest_framework.routers import DefaultRouter

from messaging.views import MessageGroupViewSet

router = DefaultRouter()
router.register(r'api/message/groups', MessageGroupViewSet, base_name='message_group')

urlpatterns = [
    path('any_url', AnyViewClass.as_view()),] + router.urls


AND IT IS DONE!
Ele vai criar os métodos de listagem, criação, edição e exclusão para o model utilizando o serializer definido no ViewSet, muito simples. Os métodos podem ser sobrescritos, assim como as views comuns. Enfim, facilitou muito o meu trabalho que era basicamente um CRUD filtrando as mensagens e grupos relacionados ao usuário.


quarta-feira, junho 07, 2017

Django OAuth server and Flask OAuth consumer with Django Outh Toolkit

Hi everyone!

I'm starting the integration of an app that will serve OAuth authentication method in a Django project. Django OAuth Toolkit appeared as a good solution, however the documentation was a little confused to me and I had a hard way of testing to understand how it works.

For this tutorial I'm using

Django==1.9.1

Flask==0.10.1
django-cors-headers==1.1.0
django-oauth-toolkit==0.10.0



The Server


First of all, create a Django project:

django-admin.py startproject my_oauth_server

After that, let's follow the steps at the oficial tutorial (assuming you have already installed the packages above):


Add oauth2_provider and corsheader in your settings.py

INSTALLED_APPS = {
    # ...
    'oauth2_provider',
    'corsheaders',
}


Add the toolkit urls:

urlpatterns = patterns(
    '',
    url(r'^admin/', include(admin.site.urls)),
    url(r'^o/', include('oauth2_provider.urls', namespace='oauth2_provider')),
    # ...
)

And the middleware:

MIDDLEWARE_CLASSES = (
    # ...
    'corsheaders.middleware.CorsMiddleware',
    # ...
)

And add this in settings:

CORS_ORIGIN_ALLOW_ALL = True

Now let's create the authentication system for our Django:

django-admin.py startapp my_authentication


By the simplest way, create the view:

from django.http.response import HttpResponse, HttpResponseRedirect
from django.shortcuts import render
from django.contrib.auth import authenticate, login, logout


# Create your views here.

def login_page(request):
    next_page = request.GET.get('next', '/')
    if request.method == 'POST':
        username = request.POST['username']
        password = request.POST['password']
        user = authenticate(username=username, password=password)
        login(request, user)
        return HttpResponseRedirect(next_page)

    return render(request, 'login.html', {'next': next_page})


def logout_page(request):
    logout(request)
    return HttpResponseRedirect('/')


def home(request):
    return HttpResponse("Logged in: %s" % request.user.is_authenticated())



Now create the template login.html:

<form method="post">    {% csrf_token %}
    <input type="hidden" name="next" value="{{ next }}" />    <p>        username
        <input type="text" name="username" />    </p>    <p>        password <input type="password" name="password" />    </p>    <button type="submit" >Login</button></form>


Actually, it's stupid simple, just to have an exemple. The "next" input receives the value from the url. This is required to were go after login. 
Finally, add your app urls:

from django.conf.urls import url, include
from django.contrib import admin

urlpatterns = [
    url(r'^admin/', admin.site.urls), 
    url(r'^o/', include('oauth2_provider.urls', namespace='oauth2_provider')),  
    url(r'^$', 'authorization_server.views.home'), 
    url(r'^accounts/login/$', 'authorization_server.views.login_page'), 
    url(r'^logout/$', 'authorization_server.views.logout_page'),]


Just run the migrate command and create an user to authorize using oauth. Create an an app at:
/o/applications. Copy the client id and client secret. Put the redirect url to /consumer at your address (in my case, the consumer will run at port 9100



The consumer

I got this exemple and applyed to my project. you can see the code bellow:

from flask import Flask
from flask import request

app = Flask(__name__)

CLIENT_ID = "Vkh2wibaQZLaxrvhYQUq8I5WWdqRLmQJXQc2YxPB"CLIENT_SECRET = "czsU5umPX2mrLqKTlLV6YNSgbPLQKxdp2WsOlWJkIYJjXP4rSrnwEwZJlAkCo7fjlCCPo3ReLMGazjeMWO5ujtjAJEOLIsTWcrE18PxwiioQMOQxTvEQQliMSCOLxEnl"REDIRECT_URI = "http://localhost:9100/consumer"
from flask import Flask

app = Flask(__name__)


# Left as an exercise to the reader.# You may want to store valid states in a database or memcache,# or perhaps cryptographically sign them and verify upon retrieval.def save_created_state(state):
    pass
def is_valid_state(state):
    return True

import requests
import requests.auth


def get_token(code):
    client_auth = requests.auth.HTTPBasicAuth(CLIENT_ID, CLIENT_SECRET)
    post_data = {"grant_type": "authorization_code",                 "code": code,                 "redirect_uri": REDIRECT_URI}
    response = requests.post("http://localhost:9000/o/token",                             auth=client_auth,                             data=post_data)
    token_json = response.json()
    return token_json["access_token"]


from flask import abort, request
@app.route('/consumer/')
def reddit_callback():
    error = request.args.get('error', '')
    if error:
        return "Error: " + error
    state = request.args.get('state', '')
    if not is_valid_state(state):
        # Uh-oh, this request wasn't started by us!        abort(403)
    code = request.args.get('code')
    # We'll change this next line in just a moment    return "got a code! %s" % code


if __name__ == '__main__':
    app.run(debug=True, port=9100)


def get_username(access_token):
    headers = {"Authorization": "bearer " + access_token}
    response = requests.get("http://localhost:9000/user_data", headers=headers)
    me_json = response.json()
    return me_json['username']



Run the flask server. Here I used the port 9000 to the server and 9100 for the consumer. The authentication url:

http://localhost:9000/o/authorize/?response_type=code&state=random_state_string&client_id=Vkh2wibaQZLaxrvhYQUq8I5WWdqRLmQJXQc2YxPB

Use the right attribute values for client id. With this you can log users in your application with oauth and consume that.