Trace http calls with python-requests

Today python-requests is the de-facto standard library for rest calls.

As everything goes on TLS, you can trace api calls with the following:

import httplib as http_client
http_client.HTTPConnection.debuglevel = 1
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.propagate = True

MySQL JSON fields on the ground!

Having to add a series of custom fields to a quite relational application, I decided to try the new JSON fields.

As of now you can:

– create json fields
– manipulate them with json_extract, json_unquote
– create generated fields from json entries

You can not:

– index json fields directly, create a generated field and index it
– retain the original json datatype (eg. string, int), as json_extract always returns strings.

Let’s start with a simple flask app:

# requirements.txt

Let’s create a simple flask app connected to a db.

import flask
import flask_sqlalchemy
from sqlalchemy.dialects.mysql import JSON

# A simple flask app connected to a db
app = flask.Flask('app')
db = flask_sqlalchemy.SQLAlchemy(app)

Add a class to the playground and create it on the db. We need sqlalchemy>=1.1 to support the JSON type!

# The model
class MyJson(db.Model):
    name = db.Column(db.String(16), primary_key=True)
    json = db.Column(JSON, nullable=True)

    def __init__(self, name, json=None): = name
        self.json = json

# Create table

Thanks to flask-sqlalchemy we can just db.session ūüėČ

# Add an entry
entry = MyJson('jon', {'do': 'it', 'now': 1})

We can now verify using a raw select that the entry is now serialized on db

# Get entry in Standard SQL
entries = db.engine.execute(['*'], from_obj=MyJson)).fetchall()
(name, json_as_string), = first_entry  # unpack result (it's just one!)
assert isinstance(json_as_string, basestring) 

A raw select to extract json fields now:

entries = db.engine.execute([name, 'json_extract(json, "$.now")'], from_obj=MyJson)).fetchall()

(name, json_now), = first_entry  # unpack result (it's just one!)
assert isinstance(json_now, basestring) 
assert json_now != entry.json['now']  # '1' != 1 

FullText Indexing IPv6 addresses with MySQL 5.7

MySQL 5.7 supports generated fields. This is particularly useful for searching the string representation of numeric stored ip addresses:

ip varbinary(16) not null,
hostname varchar(64) not null,
label varchar(64),
ip_ntoa varchar(64) generated always as (inet6_ntoa(ip)) STORED, -- generate and store fields with the address representation
fulltext key (hostname, ip_ntoa, label)

When inserting values

INSERT INTO catalog(ip,hostname,label) VALUES
(inet6_aton(''), 'localhost', 'lo'),
(inet6_aton(''), 'gimli', 'stage,ipv4'),
(inet6_aton('fdfe::5a55:caff:fefa:9089'), 'legolas', 'router,ipv6'),
(inet6_aton('fdfe::5a55:caff:fefa:9090'), 'boromir', 'router,ipv6')

you can search in OR mode with

SELECT hostname FROM catalog WHERE
  MATCH(ip_ntoa, hostname, label)
  AGAINST('9089 router');
-- returns every entry matching ANY needle
hostname: legolas
hostname: boromir

Or exact matches

SELECT hostname FROM catalog WHERE
  MATCH(ip_ntoa, hostname, label)
  AGAINST('+9089 +router' in boolean mode);
-- returns ONE entry matching ALL needles
hostname: legolas

Jboss EAP: dumping configurations as scripts!

Today I found this nice tool to create configuration scripts for Jboss EAP from existing configurations.

$  git clone
$  cd profilecloner
$  export JBOSS_HOME=/opt/jboss/
$  mvn install
$  ln -s profilecloner.jar target/profile*jar
$ ¬†./ -f save-script-here.cli ¬†–controller=$HOST –username=admin –password=secret /profile=full-ha antani-new-profile-name
$  cat save-script-here.cli


One-click remote deployment with jboss and eclipse via maven

Create a maven goal which deploy to a remote jboss eap 6.x instance is quite simple.

Just add the following to your pom.xml


Then create a maven run configuration in eclipse having:
– Base directory set to the variable ${project_path} (so that it applies to the current project)
– Goal: jboss-as:deploy

Reverse engineering included

With ipython you can write a function, like:

prompt [1]# def parse(line):
    ip, host = line.split()
    return "{host} IN PTR {ip}".format(host=host,ip=ip)

To edit our function, just use %edit and reference the line

prompt [2]# %edit 1

Once you modify the function, you cannot reference the newer code with edit, as

prompt [3]# %edit 2

just references “%edit 1” and not the newer code.

In this case we can simply recover the last code of our function with

prompt [4]# from inspect import getsourcelines
prompt [5]# getsourcelines(parse)
(['def parse(line):\n',
'    ip, host = line.split()[:2]\n',
'    return "{host} IN PTR {ip}".format(host=host,ip=ip)\n'],

cx_oracle segfault by default

Playing with the twisted adbapi connecting to oracle. Stressing a bit the application (an sftp server authenticating on oracle) I found that when the connection pool is exhausted the application crashed.

The (simple) solution is to instantiate the pool with the threaded keyword.

dbpool = adbapi.ConnectionPool("cx_Oracle", 
  threaded=True  )

Twisted is an event-based framework: all the blocking calls should be done outside the listening thread. The a(synchronous)dbapi provide separate threads for db connections. Using cx_oracle in a non thread-safe way is a reasonable cause of segfault.

So checking cx_Oracle docs we found that thread safety is off by default to gain some performance.

Oracle’s mysql.connector for python

Oracle released a pure-python mysql connector with connection pooling support.

Create a connection pool is really easy. You can try the following snippets in ipython

import mysql.connector

auth = {
"database": "test",
"user":     "user",
"password":     "secret"

# the first call instantiates the pool
mypool = mysql.connector.connect(
pool_name = "mypool",
pool_size = 3,

All the subsequent calls to connect(pool_name=”mypool”) will be managed by the pool

# this won't create another connection
#  but lend one from the pool
conn =  mysql.connector.connect(
    pool_name = "mypool",
    pool_size = 3,

# now get a cursor and play
c = conn.cursor()
c.execute("show databases")

Closing the connection will just release the connection to the pool: we’re not closing the socket!