Python import mechanism

Summarize Python Import
For beginner
Yuki Nishiwaki

Agenda
1.Module Overview
2.How to import
3.Module have some feature like single-tone
4.Module search path
1. The site.py
2. Usecase of pth file

What's module
● The entity to aggregate following entities
– Functions
– Classes
– modules(nested)
● Multiple modules are aggregated by Package
– But There is no entity in the run-time of Python
– recognized as module to have special behavior
– The module to be recognized as package should
have __path__ attribute

How is defined
Test/ ← package
├── __init__.py ← package need this file
├── non_package ← not package and module
├── test2
│ ├── __init__.py
│ └── test_module.py
└── test_module.py ← module
The directory that have __init__.py is recognized as Package.
The Module is defined as file and module name should be equal
to file name.

How to import module
import module_name
module.methodA() #-> something happen
You can call function in module_name with
module_name.methodA
You can call function in module_name with methodA
from module_name as methodA
ModuleA() #-> something happen

Implicit relative import
├── not_call.py
└── test
├── __init__.py
├── not_call.py
└── test_module.py
test/__init__.py
import not_call
$python3
>>> import test
$python2
>>> import test
Python2
Python3
As long as you can see, The path is not
included to express for relatives.
But we could import the packages based on
relatives path in python2. The reason why is
python2 search directory same as source to
execute import at first with implication.
From python3, we can't import relative
package with implication. But we can import
that with explication.
from . import not_call

The import sentence did roughly
1. Module Load
a. Try to search the module in cache
b. Read program and create module object
c. Cache it in sys.modules dictionaries
2. Bind namespace to current scope
a. Bind so name is associated with module
b. You can't use module until bind is done

The from-import did roughly
1. Module Load
a. Omit explanation due to same flow as import
2. Insert module to local variables
a. Prepare the variable
b. Insert the target (from hoge import hoge_func)
c. The target will be separated from module object

In other words
from mod import name1, name2
import mod
name1 = mod.name1
name2 = mod.name2
del mod
↑ is equals to ↓

Module is like single-tone object
● Once module is loaded, that is cached in
sys.modules dictionaries.
sys.modules = {
'module.name' : module object
}
● Since second time to try to import, Python
return the module from this dictionaries.
● We don't need to care for includ guard like C
– This cache search is done before read program file
is read file thus this mechanism contribute
performance up.

Break the behavior of single tone
● If delete the cache from sys.modules, single
tone behavior will be able to break
sys.modules
module_z
'module_z' → 4513070224 ・・
memory
4513070224
import module_z
test1.py test2.py
import module_z
break.py
import sys
del sys.modules['module_z']
import module_z
module_z
4513089454
4513089454
modue_z is test2.mozule_z
#=> False
#=> False
#=> True

How can I update cached module
● Delete cache and re-import with import sentence
– As I mentioned, This break single tone behavior
● Use reload() function
– This perform reload and store where same address.
>>> import datetime
>>> id(datetime)
4444653560
>>> reload(datetime)
<module 'datetime' from '/System/Library/lib/python2.7/lib-dynload/datetime.so'>
>>> id(datetime)
4444653560
>>> import datetime
>>> id(datetime)
4434683840
>>> del sys.modules['datetime']; import datetime
>>> id(datetime)
4434684232

Package have special behavior
>>> import test
<module 'test' from 'test/__init__.pyc'>
>>> dir(test)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
>>> locals().keys()
['test', '__warningregistry__', '__builtins__', '__package__', '__name__', '__doc__']
>>> import test.test_module
<module 'test.test_module' from 'test/test_module.py'>
>>> dir(test)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'test_module']
>>> locals().keys()
['test', '__warningregistry__', '__builtins__', '__package__', 'sys', '__name__', '__doc__']
└── test
├── __init__.py
└── test_module.py
Added module under package
to module of package

Module search path
Search Order:
● Search in ascending order of sys.path array
How to customize the module search path:
1. Add it to PYTHONPATH environment variable
2. Directory mess around with sys.path array
3. Create hoge.pth file under site-packages dir
- Add path to hoge.pth
- Write one-line code to add module to sys.modules

site.py
● This is evaluated when initialized python run-time
– If -S options is passed, site.py call will be skipped.
● This program did followings
– Add site-packages path to sys.path
– Format the relatives path to absolute path in sys.path
– Evaluate *.pth under site-packages
● If start with import, this file is recognized as Python one-line
program and exec this code.
● If not start with import, this file is recognized as the groups to
describe multiple path and add these to sys.path
– hoge.pth will be only treated by this program
● *.pth files all need to be located under site-packages dir

Environment vars associated
● $PYTHONPATH isn't handled by this program
– Before site.py is executed it's already cared for
sys.pth
● Mainly environment vars are used to assume
– What is the path of site-package
– which site-package dir is used.

Used following environment vars
$HOME :
Only mac. Use to assume site-packages path under user
home directory. In the case of Mac sometimes Python is
installed under $HOME/Library.
$PYTHONUSERBASE:
We can add user specific site-package path like
(~/.local/lib/python2.7/site-packages). The head of these
to .local dir can be specified like
(/ukinau/lib/python2.7/site-packages)

The sample of *.pth file
----ndg_httpsclients(1/2)----
How to install this sample:
● You can install this by “pip install ndg_httpsclients”
Why do we need the pth file for this package:
● The top directory(ndg) doesn't have __init__.py
● In the case of earlier version than Python3.3, we can't
load the module which doesn't have __init__.py (PEP420
Implicit Namespace package)
● Add module directly to sys.modules with one-liner code of
pth file so that we could import in the case of earlier than
it as well.

The sample of *.pth file
----ndg_httpsclients(2/2)----
1: import sys, types, os;
2: p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('ndg',));
#=> Access attr in source of call
#=> p = lib/python2.7/site-packages/ndg
3: ie = os.path.exists(os.path.join(p,'__init__.py'));
#=> ie = False (because ndg directory doesn't have __init__.py, so we add module by
this pth file)
4: m = not ie and sys.modules.setdefault('ndg',types.ModuleType('ndg'));
#=> True and sys.modules.setdefault('ndg', <module ndg>)
#=> m = <module ndg>
5: mp = (m or []) and m.__dict__.setdefault('__path__',[]);
#=> mp = m.__path__
6: (p not in mp) and mp.append(p)
#=> m.__path__.append('lib/python2.7/site-packages/ndg') <= Finish to create package

The rest of interest for import
● Inner mechanism like how to treat __path__
● Hook mechanism (PEP302)
● Imp built-in module
– Interface to be used by implementation of import
A little get off the import
● Plugin mechanism / Dynamically module load
To be continued to later.....

Python import mechanism

Recommended

More Related Content

What's hot (20)

Similar to Python import mechanism (20)

Recently uploaded (20)

Python import mechanism