1.概述
teledbx的审计是基于postgres的2个后台进程实现,audit logger process进程和audit fga worker进程(依赖pg_fga插件)。
前者主要负责粗粒度审计,下文简称audit;后者负责细粒度审计,下文简称fga。
粗粒度审计日志主要针对语句级别、用户级别、对象级别,对象操作级别,主要在sql语法解析层面进行捕捉;
细粒度粒度审计主要是针对某个表的列属性、列的值进行审计,在sql执行器进行捕捉。
补充说明:
- 两套审计规则都依赖audit_admin的审计管理员用户
- 审计管理员audit_admin、安全管理员mls_admin、数据库管理员三权分立,审计用户命名必须以 audit 开头
- audit和fga使用兼容:相互独立,可只用audit;可只用fga;可用audit和fga
--6个视图(4个粗粒度+2个细粒度)
select schemaname,viewname from pg_views where viewname like '%audit%';
schemaname | viewname
------------+------------------------------------
pg_catalog | pg_audit_stmt_conf_detail --粗粒度语句级别审计配置细节
pg_catalog | pg_audit_obj_conf_detail --粗粒度对象级别审计配置细节
pg_catalog | pg_audit_obj_def_opts_detail --粗粒度对象定义操作审计配置细节
pg_catalog | pg_audit_user_conf_detail --粗粒度用户级别审计配置细节
pg_catalog | pg_audit_fga_conf_detail --细粒度审计配置细节
pg_catalog | pg_audit_fga_policy_columns_detail --细粒度审计表列配置细节
1.1 audit(粗粒度审计)
支持4种类型的审计:Object Audit(数据库对象)、Object Default Audit(数据库元数据)、User Audit(用户审计)、Statement Audit(语句审计)
-
audit参数:
参数名 说明 值范围 默认 重要 enable_audit 是否开启audit logger子进程,确保audit_审计用户捕捉数据库对象的操作语句(PGC_SIGHUP类型参数,必须重启才能生效) [on,off] off 是 enable_audit_warning 是否打印audit 警告日志(用户级别参数) [on,off] off 否 enable_auditlogger_warning 是否将audit 警告日志写入pg_log/audit下的文件(用户级别参数) [on,off] off 否 alog_common_cache_size 本地为audit logger分配的缓存大小 [8,2097151] 64 alog_common_queue_size 每个后端用于存储 FGA 审计日志的共享内存队列大小 [8,2097151] 64 alog_file_mode 文件读写权限 alog_filename audit log文件名字 -
日志记录文件:$DATADIR/pg_log/audit/audit-*.log(cn和dn的对应目录下都有该文件,但是记录只写在cn数据目录下,dn下为空)
1.2 fga(细粒度审计)
该审计功能依赖插件pg_fga。
-
fga参数:
参数名 说明 值范围 默认 重要 enable_fga 是否开启audit fga worker子进程,确保细粒度审计的开关(PGC_SIGHUP类型参数,必须重启才能生效) [on,off] off 是 alog_fga_cache_size 本地为每个fga worker分配的缓存大小 [8,2097151] 64 否 alog_fga_queue_size 每个后端用于存储 FGA 审计日志的共享内存队列大小 [8,2097151] 64 否 -
日志记录文件:$DATADIR/pg_log/audit/audit-*.fga(执行器层面,因此大都在dn数据目录下。)
2.使用粗粒度audit
2.1 准备
在cn和dn的公共配置参数文件postgresql.conf.user配置:
enable_audit=on
enable_audit_warning=off #可选
enable_auditlogger_warning=off #可选
2.2 使用过程
登录
以audit_admin用户进行连接,执行审计规则audit all,会将后续的语句审计记录在$DATADIR/pg_log/audit/audit-*.log
$ ./psql -h 127.0.0.1 -p 8888 -d postgres -W -U audit_admin
Password for user audit_admin:
psql (PostgreSQL 10.0 TeleDBX V6)
Type "help" for help.
postgres=> audit all;
WARNING: AuditDefine Rcv: audit all;
WARNING: AuditDefine Snd: AUDIT ALL ;
AUDIT
--数据准备
\c audit_database audit_user
create table tbl_test(f1 number, f2 timestamp default now(), f3 int) ;
create table tbl_test0 as select * from tbl_test;
create view tbl_test_v as select * from tbl_test;
create materialized view tbl_test_mv as select * from tbl_test;
create view tbl_test0_v as select * from tbl_test0;
create materialized view tbl_test0_mv as select * from tbl_test0;
create schema sc_test;
create table sc_test.tbl_test(f1 number, f2 timestamp default now(), f3 int) ;
create table sc_test.tbl_test0 as select * from sc_test.tbl_test;
create view sc_test.tbl_test_v as select * from sc_test.tbl_test;
create materialized view sc_test.tbl_test_mv as select * from sc_test.tbl_test;
create view sc_test.tbl_test0_v as select * from sc_test.tbl_test0;
create materialized view sc_test.tbl_test0_mv as select * from sc_test.tbl_test0;
-- 创建审计规则语法举例
\c audit_database audit_admin
audit all; --开启所有审计
audit all by audit_user; --只通过audit_user用户去开启所有审计
audit all on default; --对元数据库开启所有审计
audit all on tbl_test; --对表开启所有审计
audit all on tbl_test_v; --对视图开启
audit all on view tbl_test_v;
audit all on tbl_test_mv; --对物化视图开启
audit all on view tbl_test_mv;
audit all on materialized view tbl_test_mv;
audit all on sc_test.tbl_test;
audit all on sc_test.tbl_test_v;
audit all on view sc_test.tbl_test_v;
audit all on sc_test.tbl_test_mv;
audit all on view sc_test.tbl_test_mv;
audit all on materialized view sc_test.tbl_test_mv;
audit all on sc_test.tbl_test0;
audit all on sc_test.tbl_test0_v;
audit all on view sc_test.tbl_test0_v;
audit all on sc_test.tbl_test0_mv;
audit all on view sc_test.tbl_test0_mv;
audit all on materialized view sc_test.tbl_test0_mv;
--4种类型:对象、用户、语句、对象定义操作
select * from pg_audit_obj_conf_detail order by auditor, object_class, object_desc, action_name, action_mode;
select * from pg_audit_user_conf_detail order by auditor, user_name, action_name, action_mode;
select * from pg_audit_stmt_conf_detail order by auditor, action_name, action_mode;
select * from pg_audit_obj_def_opts_detail order by auditor, action_name;
--noaudit的SQL语句
noaudit all WHENEVER NOT SUCCESSFUL;
noaudit all by audit_user WHENEVER NOT SUCCESSFUL;
noaudit all on default WHENEVER NOT SUCCESSFUL;
noaudit all on tbl_test WHENEVER NOT SUCCESSFUL;
noaudit all on tbl_test_v WHENEVER NOT SUCCESSFUL;
noaudit all on view tbl_test_v WHENEVER NOT SUCCESSFUL;
noaudit all on tbl_test_mv WHENEVER NOT SUCCESSFUL;
noaudit all on view tbl_test_mv WHENEVER NOT SUCCESSFUL;
noaudit all on materialized view tbl_test_mv WHENEVER NOT SUCCESSFUL;
noaudit all;
noaudit all by audit_user;
noaudit all on default;
noaudit all on tbl_test;
noaudit all on tbl_test_v;
noaudit all on view tbl_test_v;
noaudit all on tbl_test_mv;
noaudit all on view tbl_test_mv;
noaudit all on materialized view tbl_test_mv;
-- 清除审计规则SQL语句
\c audit_database audit_admin
clean unknown audit; --清除unknown的审计
clean statement audit; --清除语句审计
clean user audit by audit_user; --清除用户审计
clean object audit on table tbl_test; --清除数据库对象审计(表)
clean object audit on view tbl_test_v; --清除数据库对象审计(视图)
clean object audit on materialized view tbl_test_mv; --清除数据库对象审计(物化视图)
clean object audit; --清除数据库对象审计
clean object audit on default; --清除数据库默认元数据对象审计
clean all audit; --清除所有审计
select * from pg_audit_obj_conf_detail order by auditor, object_class, object_desc, action_name, action_mode;
select * from pg_audit_user_conf_detail order by auditor, user_name, action_name, action_mode;
select * from pg_audit_stmt_conf_detail order by auditor, action_name, action_mode;
select * from pg_audit_obj_def_opts_detail order by auditor, action_name;
2.3 验证
可以查看cn 节点下$DATADIR/pg_log/audit/audit-*.audit文件, 如以下格式
AuditOutMessage: AuditType: "Statement Audit", QueryString: "create table t1(id int);", TopCommandTag: "CREATE TABLE", DatabaseID: 13437, DatabaseName: "postgres", DatabaseUserID: 10, DatabaseUserName: "test", NodeOid: 11947, NodeName: "cn01", NodeType: "Coordinator", NodeHost: "127.0.0.1", NodePort: 8888, NodeOSUser: "test", PostgresPID: 18950, PostmasterPID: 17775, BackendStartTime: "2024-06-23 15:36:45 CST", QueryBeginTime: "2024-06-23 15:36:54 CST", QueryEndTime: "2024-06-23 15:36:55 CST", QueryIsSuccess: 1, ClientHost: "127.0.0.1", ClientHostname: "unknown", ClientPort: "36904", AppName: "psql", ObjectClassID: 1259, ObjectId: 0, ObjectSubId: 0, ObjectType: "Table", ObjectName: "t1", ActionID: 1001, ActionName: "Create Table", ExecStatus: "Exec Successfull"
核心字段解释
序 | 字段 | 说明 | 值 |
---|---|---|---|
1 | AuditType | 审计类型(4种类型) | Object Audit , Object Default Audit, User Audit, Statement Audit |
2 | QueryString | 查询语句 | sql语句,例create table t1(id int); |
3 | TopCommandTag | 对应命令的nodeTag | CREATE TABLE |
4 | DatabaseID | 数据库的oid | 13437 |
5 | DatabaseName | 数据库名字 | postgres |
6 | ExecStatus | 执行状态 | Exec Successfull或者Exec Not Successfull |
3.使用细粒度fga
3.1 准备
在cn和dn的公共配置参数文件postgresql.conf.user配置:
enable_fga=on
alog_fga_cacae_size=64kb #每一个audit worker的local buffer大小
alog_fga_queue_size=64kb #每个后端用于存储 FGA 审计日志的共享内存队列大小
3.2 使用过程
进程:在每1个cn和dn都会开启1个audit logger process和1个后台进程bgworker:audit fga worker
--tangyujie是初始化的超级用户
$ ./psql -h 127.0.0.1 -p11111 -d postgres -W -U tangyujie
--------------------------------步骤1: 创建插件&查看参数
create extension pg_fga;
show enable_fga;
--1.创建用户和数据库
CREATE USER audit_fga_user WITH SUPERUSER CREATEDB LOGIN ENCRYPTED PASSWORD 'audit_fga_user';
CREATE DATABASE audit_fga_database;
--2切换到audit_fga_database数据库
\c audit_fga_database audit_fga_user;
create extension pg_fga;
--3.切换到audit_fga_user用户,建表插入数据
\c audit_fga_database audit_fga_user
create table foo(idx bigint, str text);
insert into foo values(1, 'a');
insert into foo values(2, 'b');
insert into foo values(3, 'c');
insert into foo values(4, 'd');
select * from foo;
--------------------------------步骤2: 制定审计规则
--4.切换到 audit_admin审计管理员用户
\c audit_fga_database audit_admin
-- 4.1 增加审计规则add_policy(指定模式、表名、规则名, 列名、审计条件,模式,存储过程、是否开启审计,语句类型) add_policy(object_schema,object_name,policy_name,audit_columns,audit_condition,handler_schema,handler_module,audit_enable,statement_types,audit_column_opts),其中audit_enable必须为true才能生效,默认为true。
select add_policy(object_schema:='public', object_name:='foo', audit_columns:='idx',policy_name:='poli', audit_condition:='idx > 1');
select add_policy('public', 'foo', 'poli1','idx','idx > 1','public','hello_world','true','select');
-- 4.2 查询pg_fga插件的视图
select * from pg_audit_fga_conf;
select * from pg_audit_fga_conf_detail ;
select *from pg_audit_fga_policy_columns_detail;
--------------------------------步骤3: 生效审计规则
-- 4.3开启审计规则enable_policy(指定模式、表名、规则名),使得audit_enable为true
select enable_policy(object_schema:='public', object_name:='foo', policy_name:='poli');
select enable_policy(object_schema:='public', object_name:='foo', policy_name:='poli1');
--------------------------------其他函数
-- 4.4 删除审计规则drop_policy(指定模式、表名、规则名)。
select drop_policy(object_schema:='public', object_name:='bar', policy_name:='poli');
-- 4.5 禁用审计规则disable_policy(指定模式、表名、规则名),使得audit_enable为false
select disable_policy(object_schema:='public', object_name:='foo', policy_name:='poli1');
3.3 验证
可以查看dn 节点 $DATADIR/pg_log/audit/audit-*.fga 文件, 如以下格式
--1.以audit_fga_user登录audit_fga_database数据库产生细粒度审计日志
select idx from foo where idx > 1;
--增加以下两条日志
LogTime: "2023-11-15 16:03:30.593 CST",UserName: "audit_fga_user",DatabaseName: "audit_fga_database",ProcessPid: 190460,RemoteIpPort: "127.0.0.1:56828",NodeName: "dn01",SessionStartTime: "2023-11-15 16:03:30 CST",TableSchema: "public",TableName: "foo",PolicyName: "poli",QueryString: "SELECT idx FROM foo WHERE (idx > 1)",CommandType: "SELECT"
LogTime: "2023-11-15 16:03:30.593 CST",UserName: "audit_fga_user",DatabaseName: "audit_fga_database",ProcessPid: 190460,RemoteIpPort: "127.0.0.1:56828",NodeName: "dn01",SessionStartTime: "2023-11-15 16:03:30 CST",TableSchema: "public",TableName: "foo",PolicyName: "poli1",QueryString: "SELECT idx FROM foo WHERE (idx > 1)",CommandType: "SELECT"
4. 相关问题
- 一个sql语句满足两个审计规则,审计记录会取交集,只有一条审计记录。
- audit_admin 执行的语句会产生审计记录。
- 有cn001,cn002节点,打开cn001的粗粒度审计规则,连接cn001执行的审计语句会生效,连接cn002执行的审计语句不会生效。