Trace http calls with python-requests

Today python-requests is the de-facto standard library for rest calls.

As everything goes on TLS, you can trace api calls with the following:


import httplib as http_client
http_client.HTTPConnection.debuglevel = 1
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

Jboss EAP: dumping configurations as scripts!

Today I found this nice tool to create configuration scripts for Jboss EAP from existing configurations.

$  git clone https://github.com/tfonteyn/profilecloner.git
$  cd profilecloner
$  export JBOSS_HOME=/opt/jboss/
$  mvn install
$  ln -s profilecloner.jar target/profile*jar
$  ./profilecloner.sh -f save-script-here.cli  –controller=$HOST –username=admin –password=secret /profile=full-ha antani-new-profile-name
$  cat save-script-here.cli

Enjoy!

Using custom CXF stack with JBoss EAP 6.x

Boss delivers a set of libraries as modules.

To use them you should reference them either in:

  • MANIFEST.MF
  • jboss-deployment-structure.xml

Those libraries can be publicly or privately exposed, as per https://access.redhat.com/articles/1122333.
The list includes:

  • CXF 2.7.x
  • RestEasy
  • JBossWs


To use your custom stack as a JAXWS/RS implementation, you should specify in the jboss-deployment-structure.xml to:

  1. exclude webservices/jaxrs subsystem
  2. exclude JEE support: JAXRS is part of JEE specs and is a dependency of JEE
  3. exclude RestEasy and CXF modules
  4. eventually include, one-by-one, all the JEE dependencies you are going to use in your stack (eg. servlet)

An example app using the Jersey stack is here.
https://access.redhat.com/solutions/676573

Slides with code, highlighting with latex

While preparing the slides for EuroPython, I decided to come back to Latex after more than 10 years from my thesis. The reason is that latex modules provide code syntax highlight – something unmanageable with libreoffice.

The `minted` package provides all that with just

\begin{minted}{python}
def your_python(code):
    return here
\end{python}

You can shorten all that with a macro

\begin{pycode}
# and you can even use
# math formulas $\sum_{x=0}^{n}x
in_comments = sum(range(n+1))
\end{pycode}

The hard issue I faced was to mark code (eg. like highlighter marking) because minted allows formatting only into comments. The following didn’t work.

# we wanted the append method to be emphasized, 
a = [1, 2]
a.\emph{append}(3)
# but the output was verbatim!

After some wandering I started from the ground. Minted package uses the python script pygment to convert code into a syntax-highlighted latex page:

#pygmentize -o out.tex -f latex -l python  -O full=true out.py

So it was pygment to refuse interpreting latex formatting into the code. Luckily the latest development branch
Moreover I found in the pygment development branch, a patch allowing escape sequences into pygment.

# setting pipe (|) as an escapeinside character
def now_this(works):
    works.|\emph{append}|(1)
# run with
#pygmentize -o out.tex -f latex -l python -O full=true,escapeinside="||" out.py

After installing the last pygmentize from bitbucket, I just patched minted and enjoy the new syntax directly from latex

% set the pygmentize option in the 
%  latex page!
\begin{pycode*}{escapeinside=||}
def now_this(works):
    works.|\colorbox{yellow}{append}|(1)
\end{pycode}

Http Reverse Proxy Tunnel

Nel caso in cui il tuo computer ha completo accesso ad internet ma il server su cui stai lavorando non esce verso internet puoi settare un proxy http sul server remoto che attraverso il tunnel ssh gira le connessioni al tuo pc.

Vediamo come fare

Installiamo un semplice proxy http sul nostro pc.

Su debian:

sudo apt-get install polipo

Su RedHat:

yum install polipo

Editiamo il file di configurazione /etc/polipo/config e aggiungiamo le righe:

dnsQueryIPv6 = no
dnsNameServer = 8.8.8.8
proxyAddress = "0.0.0.0"
allowedClients = 127.0.0.1, 192.168.0.0/24

sostiture 8.8.8.8 con un dns valido.

Facciamo partire il proxy polipo:

service polipo start

Per testare che tutto è andato a buon fine proviamo il comando:

http_proxy=http://localhost:8123 wget -O - www.google.com | head

tunnel ssh

Stabiliamo una connessione verso la macchina remota in ssh con un reverse tunnel verso il nostro proxy

ssh -R 8123:localhost:8123 you@remote

usare il proxy dal server remoto

Per usare programmi sul server remoto che supportano http_proxy, bisogna configurare la variabile d’ambiente http_proxy su localhost:8123

export http_proxy=http://localhost:8123

Se tutto è andato a buon fine dovremmo poter scaricare la home page di google con il comando

wget -O - www.google.com

Generating svg from xml with python etree

Python rocks at managing xml files. Having to convert some vss shapes to odt, I just:

# vss2xhtml file.vss > file.xhtml

Then I parsed the xhtml containing multiple images with python

from xml.etree import ElementTree as ET
xml_header = """
        
        """
tree = ET.parse("file.xhtml")
# get all the images
images = tree.findall(r'//{http://www.w3.org/2000/svg}svg')

# enumerate avoids i=0, i+=1
for i, x in enumerate(images):
    destination_file = "image_%s.svg" % i
    with open(destination_file, 'w') as fd:
        fd.write(xml_header)
        fd.write(ET.tostring(x))

passing installation parameters to cpan

To fix a failing
#cpan DBD::Oracle
complaining about missing header files, you can specify the INCLUDE location like the following

#cpan <<< "
o conf makepl_arg '-h /opt/oracle-client-11/sdk/include/'
install DBD::Oracle

"

With “o conf makepl_arg” we’ re just telling which parameters to pass to
#perl Makefile.PL

Obviously each module has its own 😉

Consistent naming with iscsi + udev

If you need to export the same iscsi discs to different machines, you may want to name it consistently between various hosts.
When you setup iSCSI drives, Linux names them in the usual and unreliable /dev/sdX way (to get started with iSCSI use Michelangelo’s post http://vaunaspada.babel.it/blog/?p=596 )

Standard SCSI disks have a serial identifier: you can get it querying the /sys/block filesystem:

 #udevadm info --path=/sys/block/sdb --query=all | grep ID_SERIAL=

or checking them via their name

 #udevadm info --name=/dev/mapper/oraqdisk1 --query=all | grep DM_UUID=

And use it to identify the disk with an udev rule:

#cat > /etc/udev/rules.d/99-disk.rule <<EOF
KERNEL=="sd*", ENV{ID_SERIAL}=="the_disc_unique_id", NAME="disk0", OWNER="storage", GROUP="storage", MODE="0660"
EOF

To make sure the iscsi discs you export via tgtd have an unique serial id, you have to set it in /etc/tgt/targets.conf

<target iqn.2013-10.1.it.babel:be1>
    <backing-store /dev/mapper/VolGroup-lv_storage_0>
            scsi_id babel_testplant_s0
    </backing-store>
    <backing-store /dev/mapper/VolGroup-lv_storage_1>
        scsi_id babel_testplant_s1
    </backing-store>
    vendor_id iSCSI ACME Inc.
</target>

At this point you just have to create the following:

#cat > /etc/udev/rules.d/99-iscsi.rule <<EOF
KERNEL=="sd*", ENV{ID_SERIAL}=="babel_testplant_s0", NAME="iscsi0", OWNER="storage", GROUP="storage", MODE="0660"
KERNEL=="sd*", ENV{ID_SERIAL}=="babel_testplant_s1", NAME="iscsi1", OWNER="storage", GROUP="storage", MODE="0660"
EOF

And reload udev

#udevadm trigger

You can even do bulk-naming using globbing (aka “*” ) and environment variables (aka %E).

#
# Discover multipath devices named "oraq*" and create the corrisponding device in /dev/oraqdisk1
#    for DM_NAME check /etc/multipath.conf
SUBSYSTEM=="block", ENV{DM_NAME}=="oraq*", NAME="%E{DM_NAME}", OWNER="grid", GROUP="oinstall", MODE="0600"