Asked  7 Months ago    Answers:  5   Viewed   33 times

Reading the changes in Python 3.1, I found something... unexpected:

The sys.version_info tuple is now a named tuple:

I never heard about named tuples before, and I thought elements could either be indexed by numbers (like in tuples and lists) or by keys (like in dicts). I never expected they could be indexed both ways.

Thus, my questions are:

  • What are named tuples?
  • How to use them?
  • Why/when should I use named tuples instead of normal tuples?
  • Why/when should I use normal tuples instead of named tuples?
  • Is there any kind of "named list" (a mutable version of the named tuple)?

 Answers

36

Named tuples are basically easy-to-create, lightweight object types. Named tuple instances can be referenced using object-like variable dereferencing or the standard tuple syntax. They can be used similarly to struct or other common record types, except that they are immutable. They were added in Python 2.6 and Python 3.0, although there is a recipe for implementation in Python 2.4.

For example, it is common to represent a point as a tuple (x, y). This leads to code like the following:

pt1 = (1.0, 5.0)
pt2 = (2.5, 1.5)

from math import sqrt
line_length = sqrt((pt1[0]-pt2[0])**2 + (pt1[1]-pt2[1])**2)

Using a named tuple it becomes more readable:

from collections import namedtuple
Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)

from math import sqrt
line_length = sqrt((pt1.x-pt2.x)**2 + (pt1.y-pt2.y)**2)

However, named tuples are still backwards compatible with normal tuples, so the following will still work:

Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)

from math import sqrt
# use index referencing
line_length = sqrt((pt1[0]-pt2[0])**2 + (pt1[1]-pt2[1])**2)
 # use tuple unpacking
x1, y1 = pt1

Thus, you should use named tuples instead of tuples anywhere you think object notation will make your code more pythonic and more easily readable. I personally have started using them to represent very simple value types, particularly when passing them as parameters to functions. It makes the functions more readable, without seeing the context of the tuple packing.

Furthermore, you can also replace ordinary immutable classes that have no functions, only fields with them. You can even use your named tuple types as base classes:

class Point(namedtuple('Point', 'x y')):
    [...]

However, as with tuples, attributes in named tuples are immutable:

>>> Point = namedtuple('Point', 'x y')
>>> pt1 = Point(1.0, 5.0)
>>> pt1.x = 2.0
AttributeError: can't set attribute

If you want to be able change the values, you need another type. There is a handy recipe for mutable recordtypes which allow you to set new values to attributes.

>>> from rcdtype import *
>>> Point = recordtype('Point', 'x y')
>>> pt1 = Point(1.0, 5.0)
>>> pt1 = Point(1.0, 5.0)
>>> pt1.x = 2.0
>>> print(pt1[0])
    2.0

I am not aware of any form of "named list" that lets you add new fields, however. You may just want to use a dictionary in this situation. Named tuples can be converted to dictionaries using pt1._asdict() which returns {'x': 1.0, 'y': 5.0} and can be operated upon with all the usual dictionary functions.

As already noted, you should check the documentation for more information from which these examples were constructed.

Tuesday, June 1, 2021
 
Trott
answered 7 Months ago
58

I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.

In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.

Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.


To take two important slides from the type hinting presentation:

Why type hints?

  1. Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
  2. Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
  3. Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.

Why use static type checkers?

  • Find bugs sooner: This is self-evident, I believe.
  • The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from a behavioral aspect) you require.
  • Large teams are already running static analysis: I'm guessing this verifies the first two points.

As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.

You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.


Type hinting with mypy:

In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.

Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.

Anyways, as noted in the PEP, hinting types should generally take three forms:

  • Function annotations (PEP 3107).
  • Stub files for built-in/user modules.
  • Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)

Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).

For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.

Function annotations and special comments:

First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are generally easily inferred but others, like lists (with regard to their contents), cannot.

Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.

# Generic List, supports indexing.
from typing import List

# In this case, the type is easily inferred as type: int.
i = 0

# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = []  # type: List[str]

# Appending an int to our list
# is statically not correct.
a.append(i)

# Appending a string is fine.
a.append("i")

print(a)  # [0, 'i']

If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.

By running this with mypy, on the other hand, we get the following response:

(Python3)jimmi@jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"

Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).

Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):

def annotated(x: int, y: str) -> bool:
    return x < y

The annotated.__annotations__ attribute now has the following values:

{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}

If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:

(Python3)jimmi@jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")

Among other things, calling the function with invalid arguments will also get caught:

annotated(20, 20)

# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"

These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.

Stub files:

Stub files can be used in two different non mutually exclusive cases:

  • You need to type check a module for which you do not want to directly alter the function signatures
  • You want to write modules and have type-checking but additionally want to separate annotations from content.

What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set of three random functions in a module named randfunc.py:

def message(s):
    print(s)

def alterContents(myIterable):
    return [i for i in myIterable if i % 2 == 0]

def combine(messageFunc, itFunc):
    messageFunc("Printing the Iterable")
    a = alterContents(range(1, 20))
    return set(a)

We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed to be passed where.

Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.

# Stub for randfucn.py
from typing import Iterable, List, Set, Callable

def message(s: str) -> None: pass

def alterContents(myIterable: Iterable[int])-> List[int]: pass

def combine(
    messageFunc: Callable[[str], Any],
    itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass

The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it helps (so use them wisely).


This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.

I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).

Checkers I know of:

  • Mypy: as described here.
  • PyType: By Google, uses different notation from what I gather, probably worth a look.

Related Packages/Projects:

  • typeshed: Official Python repository housing an assortment of stub files for the standard library.

The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:

class Counter(Dict[_T, int], Generic[_T]):
        @overload
        def __init__(self) -> None: ...
        @overload
        def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
        @overload
        def __init__(self, iterable: Iterable[_T]) -> None: ...

Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.


Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:

A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.

So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.


** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.

See What are variable annotations?, as previously mentioned, for a small introduction to these.

Tuesday, June 1, 2021
 
sholsinger
answered 7 Months ago
52
from operator import attrgetter
from collections import namedtuple

Person = namedtuple('Person', 'name age score')
seq = [Person(name='nick', age=23, score=100),
       Person(name='bob', age=25, score=200)]

Sort list by name

sorted(seq, key=attrgetter('name'))

Sort list by age

sorted(seq, key=attrgetter('age'))
Wednesday, July 14, 2021
 
Novalirium
answered 5 Months ago
35

To create the subclass, you may just pass the keys of a dict directly:

MyTuple = namedtuple('MyTuple', d)

Now to create tuple instances from this dict, or any other dict with matching keys:

my_tuple = MyTuple(**d)

Beware: namedtuples compare on values only (ordered). They are designed to be a drop-in replacement for regular tuples, with named attribute access as an added feature. The field names will not be considered when making equality comparisons. It may not be what you wanted nor expected from the namedtuple type! This differs from dict equality comparisons, which do take into account the keys and also compare order agnostic.

For readers who don't really need a type which is a subclass of tuple, there probably isn't much point to use a namedtuple in the first place. If you just want to use attribute access syntax on fields, it would be simpler and easier to create namespace objects instead:

>>> from types import SimpleNamespace
>>> SimpleNamespace(**d)
namespace(a=1, b=2, c=3, d=4)

my reason for wanting to convert my dictionary to a namedtuple is so that it becomes hashable, but still generally useable like a dict

For a hashable "attrdict" like recipe, check out a frozen box:

>>> from box import Box
>>> b = Box(d, frozen_box=True)
>>> hash(b)
7686694140185755210
>>> b.a
1
>>> b["a"]
1
>>> b["a"] = 2
BoxError: Box is frozen

There may also be a frozen mapping type coming in a later version of Python, watch this draft PEP for acceptance or rejection:

PEP 603 -- Adding a frozenmap type to collections

Tuesday, July 27, 2021
 
MassiveAttack
answered 5 Months ago
25

A possible solution that might help over all is to install virtualenv. It's a python package that helps set up some of these more annoying path redirection issues as well as gives you access to use pip and easy_install to install directly into your home directory.

That's the long route.

site-packages doesn't really mean much except for a known directory where python will look. You can change you PYTHONPATH environment variable to include other directories when you try to import. I'm not sure how you're installing, but a lot of times build processes allow you to specify a --prefix to tell the build and installation process where to actually output the files. I don't know enough about your particular case to be able to tell if that applies.

Tuesday, August 17, 2021
 
d.lebedev
answered 4 Months ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :  
Share