Browse Source

Create pettus_postgresql.md

Thomas Kandler 7 years ago
parent
commit
9c2ab64dcd
1 changed files with 108 additions and 0 deletions
  1. 108
    0
      pettus_postgresql.md

+ 108
- 0
pettus_postgresql.md View File

@@ -0,0 +1,108 @@
1
+# Christophe Pettus: PostgreSQL Proficiency for Python People - PyCon 2014
2
+
3
+* `initdb` - creates the basic structures and default database which is done usually via installation
4
+* `createdb` - create new database; is a wrapper for `CREATE DATABASE ...`
5
+* always use UTF-8 for encoding (not changable later)
6
+* `pg_ctl` - start and stop PGSQL; usually wrapped in `service postgresql start/stop` (use these); always use `-m` to shutdown
7
+* `psql` - most direct way for diagnosing database problems
8
+
9
+* all data (configs, databases, etc.) is in one directory, e.g. on Windows: `C:\Program Files\PostgreSQL\9.3\data`
10
+* real data is in `/base`
11
+* `/pg_xlog` - is  not be messed with; not actually a log
12
+
13
+* two important config files `postgresql.conf` (configuration of server parameters) and `pg_hba.conf` (authentication rules)
14
+* `postgresql.conf`:
15
+	* best practice: leave everything in here as is, add `include 'postgresql.conf.include` at the bottom, uncomment in original, know where all the overrides are
16
+	* **Logging**: be generous with logs, best source to locate problems
17
+	* example:  
18
+		`log_destination = 'csvlog'` // use CSV as log format
19
+		`log_directory = 'pg_log'` // Standard  
20
+		`logging_collector = on` // Standard  
21
+		`log_filename = 'postgres-%Y-%m-%d_%H%M%S'`
22
+		`log_rotation_age = 1d` // Standard - logfile every day
23
+		`log_rotation_size = 1GB` // if log exceeds 1GB, create new file
24
+		`log_min_duration_statement = 250ms` // if something takes longer than 250ms, log it; useful for slow queries
25
+		`log_checkpoints = on` 
26
+		`log_connections = on`
27
+		`log_disconnections = on`
28
+		`log_lock_waits = on`
29
+		`log_temp_files = 0` // log all temp files
30
+		
31
+	* **Memory**: 
32
+	* *shared buffers* (means RAM used): if SysRAM < 2GB, use 20% of SysRAM; if RAM < 32GB, use 25% of RAM; if RAM > 32GB, use 8GB 
33
+	* *work_mem*: start low (32-64MB), look for 'temporary file' in logs, set to 2-3x size of largest temp file, be a hero because it may cause a huge speed-up; be careful because this memory may be used multiple times per query which means: if it's too large it will eat up the whole sys' RAM (rule of thumb: 50% of free sysRAM / active connections)
34
+	* *maintenance_work_mem*: set top 10% of sysRAM, max. 1GB, maybe higher if VACUUM problems persist
35
+	* **Checkpoints**: buffers will be flushed to disk as WAL, lot of I/O possible
36
+	* example:
37
+		`wal_buffers = 16MB`
38
+		`checkpoint_completion_target = 0 to 9` // higher is better but may take longer to reboot
39
+		`checkpoint_timeout = 10m-30m # depends on restart time; see logs for that`
40
+		`checkpoint_segments = 32 # to start`
41
+	* **Planner**: 
42
+	* `effective_io_concurrency` - set to I/O channels, e.g. 4 disk RAID = 4
43
+	* `random_page_cost` // ?? check later
44
+* `pg_hba.conf`:
45
+	* role = db object able to own other object (e.g. a table) and has privileges
46
+	* user = role + can login into system
47
+	* postgres = superuser
48
+	* by default traffic is not encrypted, turn on SSL
49
+	* only let your webservers connect
50
+	
51
+* WAL - write ahead log
52
+	* key to many PGSQL operations (replication, crashes, etc.)
53
+	* after each transaction is commited, it gets flushed to disk
54
+	* continous record of changes since last checkpoint (means: if you got a checkpoint the WAL will be flushed; when you crash, you have the checkpoint plus the WAL to replay every commit since the checkpoint was made)
55
+	* 16MB segmented files in pg_xlog; never mess with them
56
+	* `archive_command` - used for backing up WAL segments; so each time a segment is complete it can run anyting you want (e.g. rsync this segment to this other machine), important procedure for creating backups in case of a complete crash of the DB
57
+	* `synchronous_commit` - `on`: does not return on commit until WAL flush is done; `off`: returns when WAL flush is queued --> means: on `on` you will not be able to commit new data until the query is written to a WAL segment; on `off` you may loose data because the commit may not be recorded in a WAL segment
58
+
59
+* Backup and Recovery
60
+	* _backup everything at every time_
61
+	* `pg_dump`: logical snapshot of database; does not lock the DB; low load on DB; does some compression
62
+	* use `custom` format
63
+	* `pg_restore`: restores a pg_dump; quite slow (3-4x the time to write the dump; mostly because in creates new indices)
64
+	* restore using `--jobs` which says have many parallel jobs can be started; good value is: no. of cores plus one
65
+	* alternative: Point-in-Time Recovery (PITR) backup / recovery
66
+	* `select pg_start_backup(...)` - make sure WAL files are suitable for backup
67
+	* rsync WAL files somewhere (via `archive_command`)
68
+	* `select pg_start_backup()` - ??
69
+	* make sure all WAL segments are in place
70
+	* scripts to support this: [wal-e](https://github.com/wal-e/wal-e) (highly recommended, backups in Amazon S3)
71
+	* restore: copy the segments somewhere on the machine, set up `recovery.conf`, start PostgreSQL and let it recover (overall much faster than `pg_restore`)
72
+	* simple `recovery.conf` [example from MIT](https://stuff.mit.edu/afs/sipb/user/mkgray/bar/mit/postgres-8.2/share/recovery.conf.sample) or [this one](http://pgpool.projects.pgfoundry.org/pgpool-II/doc/recovery.conf.sample)
73
+	* streaming replication possible and useful if one server goes down (more in the video between 1:06:00 and 1:21:00)
74
+	* always have a disaster recovery strategy: geographical dispersion of servers, WAL archiving
75
+	* `pg_basebackup` - take a base backup of a PostgreSQL cluster; mostly used for replication but also useful for backup
76
+
77
+* Transactions, MVCC, VACUUM
78
+	* no half-done transactions possible, no transaction can see progress of other transaction
79
+	* everything runs inside a transaction, even schema changes
80
+	* this means everything is rollback-able (if transaction isn't finished)
81
+	* also means: open transactions hold resources --> keep transactions short and precise (avoid idle'ing)
82
+	* dirty reads (read during update) are prevented by Multi-Version Concurrency Control (MVCC) which basically creates different versions (called snapshots) of the database
83
+	* each snapshots is equally 'important'; there is no 'real'/'original' snapshot; the conecpt of snapshots is only applicable on these situations: reader - reader, reader - writer, writer - reader; if there is a reader - reader situation one will get blocked
84
+	* VACUUM will take out the snapshots or tuples/rows of snapshots which are present but do not belong to any transaction
85
+	* `autovacuum` runs at all times and will do this automatically
86
+	* `planner` needs statistics for executing queries effectively; `ANALYZE` will create them automatically but may have to be run after a big restore
87
+	* if `autovacuum` seems not be running because the DB gets bloated it probably is; 20-50% dead space is quite usual; one should get worried if the bloating has no stop
88
+	* use `select * from pg_stat_user_tables` for checking when last vacuum or analyze has been run; tables with plenty transactions and old vacuum/analyze timestamp may be a problem
89
+	* are there long-running/idle transactions? Is a program locking the table? Many tables? (+10000)
90
+	* solutions: reduce autovacuum sleep time, increase number of autovac. workers, do manual vacuum, fix locking sessions
91
+	
92
+* Schema Design
93
+	* normalization is important but has limits (if someone asks "is this the 3rd normal form?", give  them the look)
94
+	* pick a naming scheme and stick with it
95
+	* PostgreSQL is quite fast with joins
96
+	* use the rich typing system of PostgreSQL
97
+	* use constraints, they're cheap and fast
98
+	* use `TIMESTAMPTZ` instead of `TIMESTAMP` because the latter one will give you the timestamp somewhere (where you or the user are) opposed to a nice UTC timestamp
99
+
100
+* Indexes
101
+	* standard PostgreSQL index is a B-Tree
102
+	* you can use B-Tree on single column, multiple column (composite) and functions/expressions
103
+	* useful under these conditions: uses on of the comparison operators, selects max 10-15% of rows, is run frequently
104
+	* indexes can accelerate joins, group by considerably
105
+	
106
+Resume at 2:05:00 (to come: more indexes, optimization, updating & practical stuff)
107
+	
108
+