Python

Python Script to load or dump data into redis

Posted on

Download this gist

Install the following requirements into your virtual environemnt
pip install click redis

Usage:
To load data into redis
python redis_dump.py load [filepath]

To dump data into redis
python redis_dump.py dump [filepath] --search '*txt'

Advertisements

Python: return value from constructor

Posted on Updated on

If you want to return something other than the class instance while creating an instance of a class, use python’s __new__ method.

__new__ is the first step of instance creation. It’s called first, and is responsible for returning a new instance of your class. In contrast, __init__doesn’t return anything; it’s only responsible for initializing the instance after it’s been created.

class MyClass(object):
    def __init__(self):
        pass
    def __new__(cls):
        if i_want_to_return_something_else:
            return something_other_than_object_instance
        else:
            # this should be written else the new instance will never be returned when this class is instanciated
            return super(MyClass, cls).__new__(cls)

Pip install mysqlclient on Amazon linux gives OSError: mysql_config not found

Posted on Updated on

As suggested by https://github.com/PyMySQL/mysqlclient-python
You need to install 2 packages on you amazon linux ec2 instance:
sudo yum install python-devel mysql-devel

Name of the packages are more specific in amazon linux.
yum list | grep python3
shows you a list of python-devel packages
Install the package that meets your requirement.

sudo yum list | grep mysql
shows you a list of mysql-devel packages
Install the package that meets your requirement

In my case
sudo yum install python36-devel
yum install mysql57-devel
did the job.

Hope that helps.
Thankx for reading. Cheers!!!

Set environment variables for executing a command

Posted on

var1=foo1 var2=foo2 python run.py

Idiomatic Python

Posted on Updated on

Some handy python coding practices

Source: http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html

We often have to initialize dictionary entries before use:
This is the naïve way to do it:

navs = {}
for (foo, equity, position) in data:
    if foo not in navs:
        navs[foo] = 0
    navs[foo] += position * prices[equity]

Better way:

navs = {}
for (foo, equity, position) in data:
    navs[foo] = (navs.get(foo, 0) + position * prices[equity])

Initializing mutable dictionary values:

equities = {}
for (foo, equity) in data:
    if foo in equities:
        equities[foo].append(equity)
    else:
        equities[foo] = [equity]

dict.setdefault(key, default) does the job much more efficiently:

equities = {}
for (foo, equity) in data:
    equities.setdefault(foo, []).append(equity)

Even more better way would be

from collections import defaultdict
equities = defaultdict(list)
for (foo, equity) in data:
    equities[foo].append(equity)

Template strings

To deal with user-exposed format strings (due to security issues).

In:
from string import Template

t = Template('I am $name.')
t.substitute(name='Groot')
Out:
'I am Groot.'

Class vs Instance Variable Pitfalls

Class variables are shared across all instances of class; instance variables are unique to instance.

class Car:
    wheels = 4  # class variable
    car_instances = 0  # class variable (this one is actually useful)
    
    def __init__(self, color):
        self.color = color  # instance variable
        self.__class__.car_instances += 1  # note the extra __class__
        
a = Car('red')
b = Car('blue')

print('Cars existing:', Car.car_instances)

a.color = 'orange'
print(a.color, b.color)

a.wheels = 3  # creates a new variable a.wheels print(a.wheels, b.wheels, Car.wheels, a.__class__.wheels)
 Output:
Cars existing: 2
orange blue
3 4 4 4

Merging dicts

a = {'1': 1, '2': 2}
b = {'2': 3, '4': 4}
{**a, **b}

Output:

{‘1’: 1, ‘2’: 3, ‘4’: 4}

Crazy dict

dicts treat keys as identical if their hash values are the same and their __eq__ is True.

a = {True: 'yes', 1: 'no', 1.0: 'maybe'}

Output:

{True: ‘maybe’}

Different between tuple and list

Creating a list of 10 elements involves these steps

dis.dis('a = [1,2,3,4,5,6,7,8,9,10]')
 1 0 LOAD_CONST 0 (1)
 3 LOAD_CONST 1 (2)
 6 LOAD_CONST 2 (3)
 9 LOAD_CONST 3 (4)
 12 LOAD_CONST 4 (5)
 15 LOAD_CONST 5 (6)
 18 LOAD_CONST 6 (7)
 21 LOAD_CONST 7 (8)
 24 LOAD_CONST 8 (9)
 27 LOAD_CONST 9 (10)
 30 BUILD_LIST 10
 33 STORE_NAME 0 (a)
 36 LOAD_CONST 10 (None)
 39 RETURN_VALUE

Creating tuple involves these steps

dis.dis('a = (1,2,3,4,5,6,7,8,9,10)')
 1 0 LOAD_CONST 11 ((1, 2, 3, 4, 5, 6, 7, 8, 9, 10))
 3 STORE_NAME 0 (a)
 6 LOAD_CONST 10 (None)
 9 RETURN_VALUE

Use else to execute code after a for loop concludes

Harmful

for user in get_all_users():
    has_malformed_email_address = False
    print ('Checking {}'.format(user))
    for email_address in user.get_all_email_addresses():
        if email_is_malformed(email_address):
            has_malformed_email_address = True
            print ('Has a malformed email address!')
            break
    if not has_malformed_email_address:
        print ('All email addresses are valid!')

Idiomatic

for user in get_all_users():
    print ('Checking {}'.format(user))
    for email_address in user.get_all_email_addresses():
        if email_is_malformed(email_address):
            print ('Has a malformed email address!')
            break
    else:
        print ('All email addresses are valid!')

Avoid using ”, [], and {} as default parameters to functions

Harmful

def f(a, L=[]):
    L.append(a)
    return L
print(f(1))
print(f(2))
print(f(3))
# This will print
#
# [1]
# [1, 2]
# [1, 2, 3]

Idiomatic

def f(a, L=None):
    if L is None:
        L = []
    L.append(a)
    return L
print(f(1))
print(f(2))
print(f(3))
# This will print
# [1]
# [2]
# [3]

Use the * operator to represent the “rest” of a list

Harmful

some_list = ['a', 'b', 'c', 'd', 'e']
(first, second, rest) = some_list[0], some_list[1], some_list[2:]
(first, middle, last) = some_list[0], some_list[1:-1], some_list[-1]
(head, penultimate, last) = some_list[:-2], some_list[-2], some_list[-1]

Idiomatic

some_list = ['a', 'b', 'c', 'd', 'e']
(first, second, *rest) = some_list
(first, *middle, last) = some_list
(*head, penultimate, last) = some_list

Prefer a generator expression to a list comprehension for simple iteration

A list comprehension generates a list object and fills in all of the elements immediately. For large lists, this can be prohibitively expensive. The generator returned by a generator expression, on the other hand, generates each element “on-demand”.

Harmful:

for uppercase_name in [name.upper() for name in get_all_usernames()]:
    process_normalized_username(uppercase_name)

Idiomatic:

for uppercase_name in (name.upper() for name in get_all_usernames()):
    process_normalized_username(uppercase_name)

Shorten the try, catch, pass occurring pattern

We do this most of the time when we don’t care about the exception

try:
    foo()
except SomeException:
    pass

Do this instead

from contextlib import suppress
with suppress(SomeException):
    os.remove('i_probably_do_not_exist')

JSON to csv string in python

Posted on Updated on

from cStringIO import StringIO
import csv


def json_to_csv(data):
    output = StringIO()
    writer = csv.DictWriter(output, data[0].keys())
    writer.writeheader()
    for row in data:
        writer.writerow(row)
    return output


print (write_csv([{"name": "Dolor", "age": "23"}, {"name": "John Doe", "age": "32"}, {"age": "42", "name": "Lorem Ipsum"}], 'output.csv').getvalue())

# output
# age,name
# 23,Abiral
# 32,Lorem
# 42,asdfasd

Find max and min number from a number having the same number of digits as that number

Posted on Updated on

foo_number = 12345
min = pow(10, len(str(foo_number)) - 1) # 10000
max = pow(10, len(str(foo_number))) - 1 # 99999