In the past 2-3 weeks I’ve been working on a general purpose, cross-platform image viewer that has some built-in NSFW support. It’s called JiVE and it’s in Python 3.6 using PyQt5. A unique feature of JiVE is that it allows you to browse online images just as if they were local images.

You can find it on GitHub: https://github.com/jabbalaci/JiVE-Image-Viewer. I also wrote a detailed documentation.

Screenshots

In action:

Selecting an NSFW subreddit:

Read the docs for more info.

http://sebastianraschka.com/Articles/2014_python_2_3_key_diff.html

Problem
Python 3 on Ubuntu is sometimes a total mess. I wanted to create a Python 3 virt. env., but I got this error:

$ virtualenv -p python3 venv
Running virtualenv with interpreter /usr/bin/python3
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 8, in 
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in 
    import re
  File "/usr/lib/python3.4/re.py", line 324, in 
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in 
    from copy_reg import *
ImportError: No module named 'copy_reg'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 53, in apport_excepthook
    if not enabled():
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 24, in enabled
    import re
  File "/usr/lib/python3.4/re.py", line 324, in 
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in 
    from copy_reg import *
ImportError: No module named 'copy_reg'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 8, in 
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in 
    import re
  File "/usr/lib/python3.4/re.py", line 324, in 
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in 
    from copy_reg import *
ImportError: No module named 'copy_reg'

Awesome! :(

Solution
I found a working solution here. The following command let me create a Python 3 virt. env.:

python3 -c 'import sys; del sys.argv[0]; s = sys.argv[0]; exec(open(s).read(), {"__file__": s, "__name__": "__main__"})' `which virtualenv` -p python3 venv 

Life on Arch / Manjaro is easier. There Python 3 has been the default for years.

For creating virtual environmets, I’ve used virtualenvwrapper so far. However, Python 3.4 contains the command pyvenv that does the same thing. Since it also installs pip in the virt. env., it can replace virtualenvwrapper.

I like to store my virtual environments in a dedicated folder, separated from the project directory. virtualenvwrapper, by default, stores the virt. env.’s in the ~/.virtualenvs folder. Since I got used to this folder, I will continue to keep my virt. env.’s in this folder.

pyvenv
Say we have our project folder here: ~/python/webapps/flasky_project. Create a virt. env. for this the following way:

pyvenv ~/.virtualenvs/flasky_project

It will create a Python 3 virt. env.

virtualenv / virtualenvwrapper
For the sake of completeness, I also write here how to create virt. env.’s with virtualenv and virtualenvwrapper:

# blog post: http://goo.gl/oEdtT3

# virtualenvwrapper for Python 3 or Python 2
mkvirtualenv -p `which python3` myenv3
mkvirtualenv -p `which python2` myenv2

# virtualenv for Python 3 or Python 2
virtualenv -p python3 myproject3
virtualenv -p python2 myproject2

# When the env. is created, activate it
# and launch the command python within.
# Verify if it's the correct version.

TL; DR
If you use Python 3.4+ and you need a virt. env., use the command “pyvenv“.

Problem
You have an XML file and you want to convert it to dict or JSON.

Well, if you have a dict, you can convert it to JSON with “json.dump()“, so the real question is: how to convert an XML file to a dictionary?

Solution
There is an excellent library for this purpose called xmltodict. Its usage is very simple:

import xmltodict

# It doesn't work with Python 3! Read on for the solution!
def convert(xml_file, xml_attribs=True):
    with open(xml_file) as f:
        d = xmltodict.parse(f, xml_attribs=xml_attribs)
        return d

This worked well under Python 2.7 but I got an error under Python 3. I checked the project’s documentation and it claimed to be Python 3 compatible. What the hell?

The error message was this:

Traceback (most recent call last):
  File "/home/jabba/Dropbox/python/lib/jabbapylib2/apps/xmltodict.py", line 247, in parse
    parser.ParseFile(xml_input)
TypeError: read() did not return a bytes object (type=str)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./xml2json.py", line 27, in <module>
    print(convert(sys.argv[1]))
  File "./xml2json.py", line 17, in convert
    d = xmltodict.parse(f, xml_attribs=xml_attribs)
  File "/home/jabba/Dropbox/python/lib/jabbapylib2/apps/xmltodict.py", line 249, in parse
    parser.Parse(xml_input, True)
TypeError: '_io.TextIOWrapper' does not support the buffer interface

I even filed an issue ticket :)

After some debugging I found a hint here: you need to open the XML file in binary mode!

XML to dict (Python 2 & 3)
So the correct version that works with Python 3 too is this:

import xmltodict

def convert(xml_file, xml_attribs=True):
    with open(xml_file, "rb") as f:    # notice the "rb" mode
        d = xmltodict.parse(f, xml_attribs=xml_attribs)
        return d

XML to JSON (Python 2 & 3)
If you want JSON output:

import json
import xmltodict

def convert(xml_file, xml_attribs=True):
    with open(xml_file, "rb") as f:    # notice the "rb" mode
        d = xmltodict.parse(f, xml_attribs=xml_attribs)
        return json.dumps(d, indent=4)

Problem
I installed Manjaro Linux on one of my laptops, just to try something new. I’ve been using it for a week and I like it so far :) On my older laptop it runs smoother than Ubuntu.

Anyway, Manjaro switched to Python 3.x, that’s the default, thus “python” points to Python 3. I use Ubuntu on my other machines where Python 2 is the default. I would like to modify my scripts (at least some of them) to run on both systems.

For instance, in Python 2.x you call “raw_input”, while this function was renamed to “input” in Python 3.x.

Solution
Well, since January 2014 I start all my new scripts with this line:

from __future__ import (absolute_import, division,
                        print_function, unicode_literals)

It ensures a nice transition from Python 2 to Python 3.

To solve the “raw_input” problem, you can add these lines:

import sys

if sys.version_info >= (3, 0):
    raw_input = input

You can continue using “raw_input”, but if it’s executed with Python 3.x, “raw_input” will point to the “input” function.

Of course, the ideal solution would be to switch to Python 3, but I’m not ready for that yet :)

Update (20141228)
At the moment I’m updating my jabbapylib library. The new version will be released soon :) Since it’s a library, it should work with both Python 2 and Python 3. When I write a new script, I tend to use Python 3 these days, but a library is different. A library should support both Python 2.x and 3.x. The most widely used solution is the Six compatibility library, which is a joy to use. To solve the raw_input issue for instance, just import the line

from six.moves import input

Then — just like in Python 3 — call the function “input()” to read from the standard input. For more info. read the official docs.

Problem
On my Linux box the default Python is version 2.7. I installed bpython system-wide, thus when I launch bpython, it uses Python 2.7.

How could I have another bpython that uses Python 3? I would like to have both though. Sometimes I need Python 2, sometimes Python 3, and I would like to use the bpython shell with them.

Solution
Simply install bpython in a Python 3 virtual environment. That is, create a Python 3 environment, and install bpython in it with pip. This bpython installation will use Python 3. Then use the absolute path of this bpython executable. And no, you don’t have to activate the virt. env. each time you want to use this bpython.

In a previous post I showed how to patch virtualenvwrapper to make it suitable for creating not only Python 2 but Python 3 environments too. Using this patched virtualenvwrapper, the process looked like this on my machine:

$ cd
$ mkdir bpython_python3
$ cd bpython_python3
$ mkvirtualenv bpython_python3  # creates a Python 3 virt. env.
(bpython_python3)$ pip install bpython
(bpython_python3)$ which bpython
/home/jabba/.virtualenvs/bpython_python3/bin/bpython  # sample; will be different in your case
(bpython_python3)$ deactivate
$

I suggest adding the following aliases to your $HOME/.bashrc:

alias bpy='bpython'
alias bpy3='/home/jabba/.virtualenvs/bpython_python3/bin/bpython'

Of course, you need to customize the path for bpy3.

Now “bpy” launches bpython with Python 2, while “bpy3” starts bpython with Python 3.

Problem
Django 1.7 came out ten days ago. If you look at the updated poll tutorial, you will notice that it’s written for Python 3! So far I’ve used Python 2 only, so I thought I would redo this tutorial for two reasons: (1) learn about migrations, and (2) start using Python 3.

For my Django projects I use virtualenvwrapper to manage my virtual environments, so the first question was: how to create a Python 3 virt. env. with virtualenvwrapper?

Well, maybe my installation got corrupted somehow, but I ran into this problem:

$ mkvirtualenv -p `which python3` tmpenv
Running virtualenv with interpreter /usr/bin/python3
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 8, in <module>
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in <module>
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 53, in apport_excepthook
    if not enabled():
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 24, in enabled
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 8, in <module>
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in <module>
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'

Solution
I posted this question to reddit (see here) and I found the following solution:

First, install pip3:

sudo apt-get install python3-pip
sudo pip3 install virtualenvwrapper

Then, add the following lines to your .bashrc:

export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/workspace    # customize if needed
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv
source /usr/local/bin/virtualenvwrapper.sh

Try if it works for you. In my case, I had to edit the file “/usr/local/bin/virtualenv“. I changed its first line to

#!/usr/bin/env python3

The consequence of changing the first line was that creating Python 3 environments became the default, but at least it works now.

Create a Python 3 virt. env.:

mktmpenv

Create a Python 2 virt. env.:

mktmpenv -p `which python2`

If you use the command “virtualenv“, it will also create Python 3 environments by default. Thus, “virtualenv myproject3” will be a Python 3 env., while “virtualenv -p python2 myproject2” will be a Python 2 env.

Problem
You have Python 2 and Python 3 on the same machine. You want to install a package that requires Python 3. You cannot use the command “pip” because it will install the package as if it were written in Python 2. You want a “pip3” command.

Solution #1 (20140912)

sudo apt-get install python3-pip

Solution #2
I found the solution here.

Steps:

$ wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
$ sudo python3 get-pip.py

Now you have a “pip3” command that you can use to install Python 3 libraries.

The following entry is based on this post: Generating pseudo random text with Markov chains using Python (by Shabda Raaj).

Problem
I’ve been interested for a long time in generating “random” texts using a given corpus. A naive way is to take words randomly and drop them together but it would result in an unreadable text. The words in the generated text should come in an order that gives the impression that the text is more or less legit :)

Solution
We will use Markov chains to solve this problem. In short, a Markov chain is a stochastic process with the Markov property. By this property the changes of state of the system depend only on the current state of the system, and not additionally on the state of the system at previous steps.

The algorithm for generating pseudo random text is the following:

1. Take two consecutive words from the corpus. We will build a chain of words and the last two words of the chain represent the current state of the Markov chain.
2. Look up in the corpus all the occurrences of the last two words (current state). If they appear more than once, select one of them randomly and add the word that follows them to the end of the chain. Now the current state is updated: it consists of the 2nd word of the former tail of the chain and the new word.
3. Repeat the previous step until you reach the desired length of the generated text.

When reading and splitting up a corpus to words, don’t remove commas, punctuations, etc. This way you can get a more realistic text.

Example
Let’s see this text:

A is the father of B.
C is the father of A.

From this we can build the following dictionary:

{('A', 'is'): ['the'],
 ('B.', 'C'): ['is'],
 ('C', 'is'): ['the'],
 ('father', 'of'): ['B.', 'A.'],
 ('is', 'the'): ['father', 'father'],
 ('of', 'B.'): ['C'],
 ('the', 'father'): ['of', 'of']}

The key is a tuple of two consecutive words. The value is a list of words that follow the two words in the key in the corpus. The value is a multiset, i.e. duplications are allowed. This way, if a word appears several times after the key, it will be selected with a higher probability.

Let’s start the generated sentence with “A is“. “A is” is followed by “the” (“A is the“). “is the” is followed by “father” (“A is the father“). “the father” is followed by “of” (“A is the father of“). At “father of” we have a choice: let’s pick “A” for instance. The end result is: “A is the father of A.“.

Python code
This is a basic version of the algorithm. Since the input corpus can be a UTF-8 file, I wrote it in Python 3 to suffer less with Unicode.

#!/usr/bin/env python3
# encoding: utf-8

import sys
from pprint import pprint
from random import choice

EOS = ['.', '?', '!']


def build_dict(words):
    """
    Build a dictionary from the words.

    (word1, word2) => [w1, w2, ...]  # key: tuple; value: list
    """
    d = {}
    for i, word in enumerate(words):
        try:
            first, second, third = words[i], words[i+1], words[i+2]
        except IndexError:
            break
        key = (first, second)
        if key not in d:
            d[key] = []
        #
        d[key].append(third)

    return d


def generate_sentence(d):
    li = [key for key in d.keys() if key[0][0].isupper()]
    key = choice(li)

    li = []
    first, second = key
    li.append(first)
    li.append(second)
    while True:
        try:
            third = choice(d[key])
        except KeyError:
            break
        li.append(third)
        if third[-1] in EOS:
            break
        # else
        key = (second, third)
        first, second = key

    return ' '.join(li)


def main():
    fname = sys.argv[1]
    with open(fname, "rt", encoding="utf-8") as f:
        text = f.read()

    words = text.split()
    d = build_dict(words)
    pprint(d)
    print()
    sent = generate_sentence(d)
    print(sent)
    if sent in text:
        print('# existing sentence :(')

####################

if __name__ == "__main__":
    if len(sys.argv) == 1:
        print("Error: provide an input corpus file.")
        sys.exit(1)
    # else
    main()

Tips
Try to choose a long corpus to work with.

In our version the current state consists of two words. If you decide to put more words (3 for instance) in the current state, then the text will look less random, but also, it will look less gibberish (see also this gist).

Links

• Generating pseudo random text with Markov chains using Python (this present post is based on this)
• Mark V. Shaney at Your Service
• Generate random words based on markov chains rather than random sentences

Blog Stats

• 1,659,944 hits

Random Post

Recent Posts

• Get the value of PI without a constant
• In a graph, find the largest complete subgraph
• How to solve a two-variable equation with Sympy
• Autoformat your Python code with Ruff
• Play a sound file from memory

Archives

• December 2025 (1)
• December 2024 (2)
• March 2024 (1)
• February 2024 (3)
• June 2023 (1)
• December 2022 (1)
• May 2022 (1)
• March 2022 (1)
• February 2022 (2)
• October 2020 (3)
• September 2020 (2)
• July 2020 (1)
• December 2019 (1)
• November 2019 (7)
• June 2019 (2)
• May 2019 (1)
• February 2019 (1)
• January 2019 (1)
• August 2018 (1)
• July 2018 (4)
• June 2018 (5)
• May 2018 (1)
• March 2018 (2)
• February 2018 (1)
• January 2018 (2)
• December 2017 (2)
• October 2017 (1)
• August 2017 (1)
• May 2017 (2)
• April 2017 (1)
• March 2017 (1)
• February 2017 (3)
• January 2017 (5)
• December 2016 (3)
• October 2016 (2)
• September 2016 (1)
• August 2016 (4)
• July 2016 (3)
• June 2016 (2)
• May 2016 (2)
• April 2016 (1)
• March 2016 (4)
• February 2016 (1)
• January 2016 (4)
• December 2015 (5)
• November 2015 (8)
• October 2015 (2)
• September 2015 (3)
• August 2015 (6)
• July 2015 (4)
• June 2015 (1)
• May 2015 (3)
• April 2015 (2)
• February 2015 (1)
• January 2015 (6)
• December 2014 (5)
• November 2014 (2)
• October 2014 (1)
• September 2014 (1)
• August 2014 (14)
• July 2014 (3)
• June 2014 (6)
• May 2014 (3)
• April 2014 (2)
• March 2014 (4)
• February 2014 (3)
• January 2014 (19)
• December 2013 (8)
• November 2013 (9)
• October 2013 (4)
• September 2013 (10)
• August 2013 (16)
• July 2013 (4)
• June 2013 (7)
• May 2013 (7)
• April 2013 (3)
• March 2013 (12)
• February 2013 (2)
• January 2013 (10)
• December 2012 (18)
• November 2012 (3)
• October 2012 (4)
• September 2012 (5)
• August 2012 (1)
• July 2012 (1)
• May 2012 (8)
• April 2012 (9)
• March 2012 (17)
• February 2012 (3)
• January 2012 (8)
• November 2011 (11)
• October 2011 (7)
• September 2011 (17)
• August 2011 (5)
• June 2011 (1)
• May 2011 (7)
• April 2011 (21)
• March 2011 (21)
• February 2011 (7)
• December 2010 (4)
• November 2010 (1)
• October 2010 (16)
• September 2010 (15)

Meta

• Create account
• Log in
• Entries feed
• Comments feed
• WordPress.com