# Bisama POS Operations & Process Documentation

**Version:** 1.1  
**Date:** January 25, 2026  
**Audience:** System Administrators, Operations Managers, DevOps, IT Support  
**Status:** Production

---

## TABLE OF CONTENTS

1. [Executive Summary](#executive-summary)
2. [System Architecture Overview](#system-architecture-overview)
3. [Operational Procedures](#operational-procedures)
4. [Tenant & Branch Management](#tenant--branch-management)
5. [Backup & Disaster Recovery](#backup--disaster-recovery)
6. [Database Management](#database-management)
7. [Financial Period Management](#financial-period-management)
8. [Monitoring & Alerting](#monitoring--alerting)
9. [Incident Response](#incident-response)
10. [Change Management](#change-management)
11. [Security Operations](#security-operations)
12. [Troubleshooting Guide](#troubleshooting-guide)

---

## EXECUTIVE SUMMARY

### System Overview

**Bisama POS** is a multi-tenant, cloud-native POS system serving Ghana-based businesses with:
- **Multi-tenancy:** Unlimited independent businesses on single platform
- **Accounting:** Automatic double-entry bookkeeping (IAS 18/IFRS 15 compliant)
- **Tax Integration:** Ghana tax rules (VAT 15%, NHIL 2.5%, GETFUND 2.5%)
- **Data Isolation:** Complete tenant separation at database/application level
- **24/7 Availability:** Uptime target: 99.9%

### Operational Responsibilities

```
System Admin
├─ Tenant creation/management
├─ Backup/restore procedures
├─ Database maintenance
├─ User account administration
└─ Security patches

DevOps Engineer
├─ Infrastructure provisioning
├─ Application deployment
├─ Monitoring & alerting
├─ Performance optimization
└─ Disaster recovery

Database Administrator
├─ Database optimization
├─ Index maintenance
├─ Query performance tuning
├─ Data integrity verification
└─ Archive management
```

---

## SYSTEM ARCHITECTURE OVERVIEW

### Technical Stack

```
Application Layer:
├─ Framework: Laravel 11 (PHP 8.2+)
├─ Web Server: Nginx/Apache
├─ Cache: Redis 7.0+
├─ Queue: Redis Queues
└─ Storage: Local/S3

Database Layer:
├─ System DB: MySQL 8.0+ (Central)
├─ Tenant DB: MySQL 8.0+ (Per tenant)
├─ Backup: SQL dumps + binary logs
└─ Replication: Master-Slave (optional)

Frontend:
├─ Template Engine: Blade
├─ CSS Framework: Tailwind CSS
├─ JavaScript: Alpine.js
└─ Real-time: WebSockets (optional)
```

### System Database Schema

```
SYSTEM DATABASE (shared across all tenants):
├─ users (system admin accounts)
├─ tenants (business accounts)
├─ tenant_branches (multi-branch per tenant)
├─ activity_logs (system audit trail)
├─ backups (backup metadata)
└─ settings (global configuration)

TENANT DATABASE (isolated per tenant):
├─ sales (retail + wholesale)
├─ sale_items (line items)
├─ payments (receipt collection)
├─ customers (client master)
├─ inv_products (product catalog)
├─ inv_movements (stock tracking)
├─ inv_stock_levels (current quantities)
├─ acc_accounts (chart of accounts)
├─ acc_journals (journal entries)
├─ acc_postings (actual postings)
├─ returns (returns & exchanges)
├─ transfers (inter-branch transfers)
├─ tax_profiles (Ghana tax config)
└─ fiscal_periods (monthly close dates)
```

### Multi-Tenancy Architecture

```
Internet Traffic
    ↓
Load Balancer (HTTPS/443)
    ↓
Nginx Reverse Proxy
    ├─ Extracts domain (e.g., acmebusiness.bisamapos.com)
    └─ Routes to PHP-FPM
         ↓
    TenantMiddleware
    ├─ Looks up tenant_id from domain
    ├─ Sets app('tenant') = Tenant instance
    └─ Applies global scope to all queries
         ↓
    Application Code
    ├─ All queries filtered by tenant_id
    ├─ Separate database per tenant (optional)
    └─ Complete data isolation guaranteed
         ↓
    Response (HTTPS encrypted)
    ├─ Return to correct tenant
    └─ No data leakage
```

---

## OPERATIONAL PROCEDURES

### Daily Operations Checklist

```
MORNING (7:00 AM):
☐ Check system status page
☐ Verify all services running (PHP-FPM, MySQL, Redis, Nginx)
☐ Check disk space (> 20% free required)
☐ Review overnight error logs
☐ Verify database replication lag (if applicable)

THROUGHOUT DAY:
☐ Monitor response times (< 2 seconds target)
☐ Monitor error rate (< 0.5% target)
☐ Monitor active connections (< 100 concurrent)
☐ Check Redis memory usage (< 80% capacity)
☐ Monitor queue job processing time

EVENING (5:00 PM):
☐ Run daily backup verification
☐ Review user activity for suspicious patterns
☐ Check disk space again
☐ Verify tomorrow's scheduled maintenance window
☐ Prepare incident response team

NIGHT (11:00 PM):
☐ Begin maintenance window (if scheduled)
☐ Run automated backups
☐ Run database optimization (ANALYZE/OPTIMIZE tables)
☐ Clear old logs (> 30 days)
☐ Verify backup integrity
```

### Weekly Operations Checklist

```
MONDAY (Start of week):
☐ Review previous week's incidents
☐ Update incident report dashboard
☐ Verify all tenant instances healthy
☐ Check license/compliance status

WEDNESDAY (Mid-week):
☐ Performance analysis (slow queries, high CPU)
☐ Security patches check (CVEs)
☐ Review user access patterns
☐ Test backup restoration

FRIDAY (End of week):
☐ Generate weekly operations report
☐ Review capacity planning
☐ Update documentation
☐ Plan next week's maintenance
☐ Communicate status to stakeholders
```

### Monthly Operations Checklist

```
FIRST WEEK:
☐ Month-end financial close support
☐ Verify all accounting postings completed
☐ Trial balance verification across all tenants
☐ Tax calculation audit
☐ Generate financial compliance report

SECOND WEEK:
☐ Database maintenance window
  ├─ Full index rebuild
  ├─ Table optimization (OPTIMIZE TABLE)
  ├─ Update table statistics
  └─ Analyze slow query logs
☐ Disaster recovery drill
  ├─ Restore from backup to test environment
  ├─ Verify data completeness
  └─ Test failover procedures
☐ Security audit
  ├─ Review access logs
  ├─ Verify firewall rules
  └─ Check SSL certificates expiration

THIRD WEEK:
☐ Capacity planning review
  ├─ Current disk usage trends
  ├─ Database size growth
  ├─ User growth analysis
  └─ Forecast 6-month needs
☐ Performance baseline update
  ├─ Query performance analysis
  ├─ Cache hit rate review
  └─ Update expected latency baselines

FOURTH WEEK:
☐ Planning for next month
  ├─ Schedule major updates
  ├─ Plan infrastructure changes
  └─ Prepare user communications

### Payroll & HR Operations (New)

```
PAYROLL CYCLE (Monthly):
☐ Verify PAYE/SSNIT rates in App Settings → PAYE/SSNIT
☐ Run "Automate Payroll" for current month
☐ Review draft payslips (gross, PAYE, SSNIT, net)
☐ Post payroll to ledger (expense vs. payable)
☐ Export payslips and archive copy

ATTENDANCE INGESTION (Weekly):
☐ Upload attendance CSV/Excel via Attendance → Upload
☐ Verify validation summary (accepted/rejected rows)
☐ Generate Attendance Report for the week
☐ Sync discrepancies with HR before payroll run

MESSAGING (Customer/Employee):
☐ Confirm SMS sender ID and credits
☐ Send bulk notifications after hours to reduce load
☐ Monitor delivery errors and throttle if >5% failures

TRUCK & DRIVER SALES RECONCILIATION (Daily):
☐ Reconcile truck sales vs. truck payments and truck returns
☐ Review driver sales report and deposit slips
☐ Investigate unpaid invoices using truck_payments reports
```
```

---

## TENANT & BRANCH MANAGEMENT

### Onboarding New Tenant

**Duration:** 1-2 hours  
**Performed by:** System Administrator

#### Step 1: Create Tenant Record

```sql
-- In System Database
INSERT INTO tenants (
    name,
    domain,
    status,
    subscription_tier,
    created_at
) VALUES (
    'ACME Trading Company',
    'acme.bisamapos.com',
    'active',
    'professional',
    NOW()
);

-- Get tenant ID
SELECT id FROM tenants WHERE domain = 'acme.bisamapos.com';
-- Result: tenant_id = 15
```

#### Step 2: Create Database

```bash
# Create separate database for tenant (optional but recommended)
mysql -u root -p << EOF
CREATE DATABASE bisamapos_tenant_15 CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
GRANT ALL PRIVILEGES ON bisamapos_tenant_15.* TO 'bisamapos_user'@'localhost';
FLUSH PRIVILEGES;
EOF
```

#### Step 3: Initialize Tenant Database

```bash
# Run migrations for new tenant
php artisan migrate --database=tenant_15 --path=database/migrations/tenant

# Seed initial data
php artisan db:seed --database=tenant_15 TenantSeeder
```

#### Step 4: Create Main Branch

```
Navigation: System Admin Panel → Tenant Management → tenant_id=15 → Branches → Add Branch

Branch Details:
├─ Branch Name: "Accra Head Office"
├─ Branch Code: "ACC-01"
├─ Address: "123 High Street, Accra"
├─ Phone: "024 1234567"
├─ Email: "accra@acme.com.gh"
└─ Status: Active

System Auto-Creates:
├─ Chart of Accounts (17 accounts per IAS)
├─ Fiscal Period (Current month)
├─ Default inventory location
└─ Default cost center
```

#### Step 5: Create Admin User

```
Navigation: System Admin Panel → User Management → Add User

User Details:
├─ Email: "admin@acme.com.gh"
├─ Name: "John Owusu"
├─ Role: "Director" (or Manager for branch admin)
├─ Branch: "Accra Head Office"
├─ Status: Active
└─ Set initial password

System Auto-Sends:
├─ Welcome email with credentials
├─ Login instructions
└─ First-time setup guide
```

#### Step 6: Configure Tax Profile

```
Navigation: System Admin Panel → Tenant Settings → Tax Configuration

Ghana Tax Settings:
├─ Tax Scheme: "Standard Scheme"
├─ VAT Rate: 15%
├─ NHIL Rate: 2.5%
├─ GETFUND Rate: 2.5%
├─ Tax Registration Number: "[GRA TIN]"
├─ Fiscal Year End: December 31
└─ First Tax Period: January

System Auto-Calculates:
├─ Monthly posting dates
├─ Quarterly declaration deadlines
└─ Annual filing reminders
```

#### Step 7: Verification

```bash
# Test tenant access
curl -H "Host: acme.bisamapos.com" https://localhost/api/health

# Response expected:
{
  "status": "ok",
  "tenant_id": 15,
  "name": "ACME Trading Company",
  "database": "bisamapos_tenant_15"
}

# Test login
POST /login
  email: admin@acme.com.gh
  password: [initial password]

# Verify database connection
SELECT COUNT(*) FROM acc_accounts WHERE tenant_id = 15;
-- Should return 17 (default accounts)
```

### Adding New Branch to Existing Tenant

**Duration:** 30 minutes  
**Performed by:** Director/System Admin

```
Navigation: Accounting → Setup → Manage Branches → Add Branch

Step 1: Enter Branch Details
├─ Branch Name: "Tema Branch"
├─ Branch Code: "TEM-02"
├─ Address: "456 Port Road, Tema"
└─ Status: Active

Step 2: System Auto-Creates
├─ Chart of Accounts for branch
├─ Inventory location code
├─ Cost center
├─ Fiscal period entry
└─ Default GL accounts

Step 3: Add Branch Staff
├─ Assign Manager (Role: Manager)
├─ Assign Cashiers (Role: User)
├─ Assign Accountant (Role: Manager)
└─ Set permissions by staff member

Step 4: Configuration
├─ POS terminals: Map to terminals
├─ Printers: Configure receipt printers
├─ Payment methods: Enable cash/card/mobile
└─ Inventory: Set initial stock levels

Step 5: Verification
├─ Login as Manager → Should see only Tema branch
├─ Create test sale
├─ Verify posting to correct GL accounts
└─ Check inventory tracking
```

### Offboarding Tenant

**Duration:** 2-3 hours  
**Performed by:** System Administrator  
**⚠️ Caution:** This process is permanent and cannot be undone

```
Step 1: Backup Tenant Data
  ├─ Export all financial reports
  ├─ Export customer list
  ├─ Full database backup (additional copy)
  └─ Archive to separate storage

Step 2: Notify Tenant
  ├─ Send 30-day termination notice
  ├─ Ensure final month reconciled
  ├─ Get final approval to delete

Step 3: Deactivate Tenant
  UPDATE tenants SET status = 'inactive' WHERE id = 15;
  
  Effects:
  ├─ Users cannot login
  ├─ API endpoints return 403
  ├─ Read-only access available (30 days)
  └─ Dashboard displays "Account Suspended"

Step 4: Final Backup (after 30 days)
  ├─ Create final archive
  ├─ Verify completeness
  └─ Store for compliance (7 years)

Step 5: Delete Tenant
  -- After 30-day grace period
  DELETE FROM tenants WHERE id = 15;
  DROP DATABASE bisamapos_tenant_15;
  
  This:
  ├─ Removes tenant from system
  ├─ Makes domain available for reuse
  ├─ Purges all data
  └─ Frees database resources
```

---

## BACKUP & DISASTER RECOVERY

### Backup Strategy

```
BACKUP TIERS:

Tier 1: System Database (Hourly)
├─ Location: /backups/system/
├─ Frequency: Every 1 hour
├─ Retention: 24 hours
├─ Size: ~100 MB each
└─ Automated: YES (cron job)

Tier 2: Tenant Databases (Daily)
├─ Location: /backups/tenants/
├─ Frequency: Every 24 hours (11 PM)
├─ Retention: 30 days
├─ Size: Varies by tenant (avg 500 MB)
└─ Automated: YES

Tier 3: Full System Backup (Weekly)
├─ Location: /backups/weekly/
├─ Frequency: Every Sunday midnight
├─ Retention: 90 days
├─ Size: Complete copy (~10 GB+)
└─ Automated: YES

Tier 4: Compliance Archive (Monthly)
├─ Location: /archive/compliance/
├─ Frequency: Every month (1st)
├─ Retention: 7 years
├─ Size: Compressed archive (~2 GB)
└─ Automated: YES
```

### Automated Backup Procedure

```bash
#!/bin/bash
# /scripts/backup-daily.sh (runs nightly at 11 PM)

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/tenants/$TIMESTAMP"
LOG_FILE="/var/log/backup-$TIMESTAMP.log"

# Create backup directory
mkdir -p $BACKUP_DIR

# Backup system database
mysqldump -u root -p$DB_PASS --single-transaction \
  system_db > $BACKUP_DIR/system_db.sql 2>>$LOG_FILE

# Backup all tenant databases
for TENANT_ID in $(mysql -u root -p$DB_PASS -e \
  "SELECT id FROM system_db.tenants WHERE status='active';" \
  -N); do
  
  TENANT_DB="bisamapos_tenant_$TENANT_ID"
  
  mysqldump -u root -p$DB_PASS --single-transaction \
    $TENANT_DB > $BACKUP_DIR/${TENANT_DB}.sql 2>>$LOG_FILE
done

# Compress backup
tar -czf $BACKUP_DIR.tar.gz $BACKUP_DIR/
rm -rf $BACKUP_DIR

# Verify backup
if tar -tzf $BACKUP_DIR.tar.gz > /dev/null; then
  echo "Backup successful: $BACKUP_DIR.tar.gz" >> $LOG_FILE
  # Upload to S3
  aws s3 cp $BACKUP_DIR.tar.gz s3://bisamapos-backups/
else
  echo "Backup verification failed!" >> $LOG_FILE
  # Alert ops team
  mail -s "BACKUP FAILED" ops@bisamapos.com < $LOG_FILE
fi

# Cleanup old backups (keep 30 days)
find /backups/tenants/ -name "*.tar.gz" -mtime +30 -delete
```

### Manual Backup Procedure

```bash
# Manual full backup (if automated fails)

# Step 1: Stop non-critical services
systemctl stop redis
systemctl stop queue:work

# Step 2: Dump all databases
mysqldump -u root -p --all-databases --single-transaction \
  --quick --lock-tables=false \
  > /backups/manual/full_dump_$(date +%Y%m%d_%H%M%S).sql

# Step 3: Backup file storage
rsync -av /storage/uploads/ /backups/manual/uploads/

# Step 4: Compress
tar -czf /backups/manual/bisamapos_complete_$(date +%Y%m%d).tar.gz \
  /backups/manual/*.sql \
  /backups/manual/uploads/

# Step 5: Verify
tar -tzf /backups/manual/bisamapos_complete_*.tar.gz | head

# Step 6: Resume services
systemctl start redis
systemctl start queue:work

# Step 7: Report
ls -lh /backups/manual/ | mail -s "Manual Backup Complete" ops@bisamapos.com
```

### Restore Procedure

**⚠️ Warning:** Restores overwrite existing data

#### Restore Specific Tenant

```bash
# Step 1: Get backup file
BACKUP_FILE="/backups/tenants/20260124_110000/bisamapos_tenant_15.sql"

# Step 2: Create temporary database
mysql -u root -p << EOF
DROP DATABASE IF EXISTS bisamapos_tenant_15_restore;
CREATE DATABASE bisamapos_tenant_15_restore CHARACTER SET utf8mb4;
EOF

# Step 3: Restore data to temporary database
mysql -u root -p bisamapos_tenant_15_restore < $BACKUP_FILE

# Step 4: Run consistency checks
mysql -u root -p bisamapos_tenant_15_restore << EOF
-- Check account balance
SELECT SUM(CASE WHEN type='debit' THEN amount ELSE -amount END) as balance
FROM acc_postings
WHERE status = 'posted';
-- Should be 0 or very close (rounding)

-- Check inventory
SELECT COUNT(*) as stock_records FROM inv_stock_levels;
-- Compare with expected count
EOF

# Step 5: If verified, switch databases
mysql -u root -p << EOF
DROP DATABASE bisamapos_tenant_15;
RENAME TABLE bisamapos_tenant_15_restore.* TO bisamapos_tenant_15.*;
EOF

# Step 6: Run migrations (if needed)
php artisan migrate --database=tenant_15

# Step 7: Test access
curl -H "Host: acme.bisamapos.com" https://localhost/api/health
```

#### Restore Entire System

```bash
# For complete system failure/rebuild

# Step 1: Extract backup
tar -xzf /backups/weekly/bisamapos_complete_20260117.tar.gz

# Step 2: Stop application
systemctl stop nginx php-fpm redis

# Step 3: Restore system database
mysql -u root -p < system_db.sql

# Step 4: Create all tenant databases
for SQL_FILE in bisamapos_tenant_*.sql; do
  TENANT_ID=$(echo $SQL_FILE | sed 's/[^0-9]//g')
  DB_NAME="bisamapos_tenant_$TENANT_ID"
  
  mysql -u root -p << EOF
  CREATE DATABASE $DB_NAME CHARACTER SET utf8mb4;
  EOF
  
  mysql -u root -p $DB_NAME < $SQL_FILE
done

# Step 5: Restore application files
rsync -av ./app/ /var/www/bisamapos/app/
rsync -av ./config/ /var/www/bisamapos/config/
rsync -av ./storage/ /var/www/bisamapos/storage/

# Step 6: Clear cache
php artisan cache:clear
php artisan config:clear

# Step 7: Start services
systemctl start redis php-fpm nginx

# Step 8: Verify
php artisan tinker
>>> Tenant::count() // Should return # of tenants
```

---

## DATABASE MANAGEMENT

### Database Optimization

#### Monthly Maintenance Window

```bash
#!/bin/bash
# /scripts/optimize-database.sh

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
LOG="/var/log/db-optimize-$TIMESTAMP.log"

echo "Database optimization started: $TIMESTAMP" >> $LOG

# Analyze all tables
mysql -u root -p -e "ANALYZE TABLE ALL;" >> $LOG 2>&1

# Optimize all tables
mysql -u root -p -e "OPTIMIZE TABLE ALL;" >> $LOG 2>&1

# Update table statistics
mysql -u root -p -e "ANALYZE TABLE ALL;" >> $LOG 2>&1

# Rebuild indexes
mysql -u root -p << EOF >> $LOG 2>&1
-- Identify unused indexes
SELECT OBJECT_SCHEMA, OBJECT_NAME, INDEX_NAME
FROM performance_schema.table_io_waits_summary_by_index_usage
WHERE OBJECT_SCHEMA != 'mysql'
  AND COUNT_STAR = 0
  AND INDEX_NAME != 'PRIMARY'
LIMIT 20;

-- Drop unused indexes (review before dropping!)
-- ALTER TABLE table_name DROP INDEX index_name;
EOF

echo "Optimization completed: $(date +%Y%m%d_%H%M%S)" >> $LOG
```

#### Index Strategy

```sql
-- Critical indexes that must be maintained

-- System Database
CREATE INDEX idx_tenants_domain ON tenants(domain);
CREATE INDEX idx_tenants_status ON tenants(status);
CREATE INDEX idx_users_email ON users(email);

-- Tenant Databases
CREATE INDEX idx_sales_date ON sales(created_at);
CREATE INDEX idx_sales_customer_id ON sales(customer_id);
CREATE INDEX idx_sales_posted ON sales(posting_status);

CREATE INDEX idx_payments_sale_id ON payments(sale_id);
CREATE INDEX idx_payments_date ON payments(created_at);

CREATE INDEX idx_journals_date ON acc_journals(entry_date);
CREATE INDEX idx_postings_account ON acc_postings(account_id);
CREATE INDEX idx_postings_period ON acc_postings(fiscal_period_id);

CREATE INDEX idx_stock_levels_product ON inv_stock_levels(product_id);
CREATE INDEX idx_movements_date ON inv_movements(created_at);
```

### Monitoring Database Health

```sql
-- Check active connections
SHOW PROCESSLIST;
-- Alert if > 100 connections

-- Check table sizes
SELECT 
    TABLE_NAME,
    ROUND(((data_length + index_length) / 1024 / 1024), 2) as size_mb
FROM information_schema.TABLES
WHERE TABLE_SCHEMA NOT IN ('mysql', 'information_schema', 'performance_schema')
ORDER BY (data_length + index_length) DESC;

-- Check slow queries (if slow query log enabled)
SELECT 
    query_time,
    lock_time,
    rows_sent,
    rows_examined,
    sql_text
FROM mysql.slow_log
WHERE start_time > DATE_SUB(NOW(), INTERVAL 1 HOUR)
ORDER BY query_time DESC
LIMIT 10;

-- Check replication lag (if replicated)
SHOW SLAVE STATUS\G
-- Seconds_Behind_Master should be < 1
```

### Archiving Old Data

```bash
# Archive transactions older than 1 year

# Step 1: Export old data
mysql -u root -p << EOF
SELECT * INTO OUTFILE '/tmp/sales_archive_2024.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
FROM sales
WHERE YEAR(created_at) = 2024;
EOF

# Step 2: Compress archive
gzip /tmp/sales_archive_2024.csv

# Step 3: Upload to S3
aws s3 cp /tmp/sales_archive_2024.csv.gz \
  s3://bisamapos-archives/2024/

# Step 4: Delete from live database
DELETE FROM sales WHERE YEAR(created_at) = 2024;

# Step 5: Optimize after deletion
OPTIMIZE TABLE sales;
```

---

## FINANCIAL PERIOD MANAGEMENT

### Monthly Financial Close Procedure

**Duration:** 2-4 hours  
**Performed by:** Accounting Manager  
**Schedule:** Last day of every month (11 PM)

#### Phase 1: Pre-Close Verification (2 PM)

```
Checklist:
☐ All sales entered for the month
  └─ Check: MAX(created_at) FROM sales >= month start
  
☐ All payments recorded
  └─ Check: All invoices either paid or AR aging < 90 days
  
☐ All expenses entered
  └─ Check: Count of expense entries vs. expected
  
☐ All inventory movements recorded
  └─ Check: No pending stock transfers
  
☐ Trial balance (before posting)
  └─ Check: Total debits = Total credits (within 0.01)
```

#### Phase 2: Automated Posting (3 PM)

```bash
# Bisama POS auto-posts all unposted transactions

php artisan accounting:post-unposted --period=20260131

Output:
  Processing 1,234 unposted transactions
  ├─ Sales invoices: 892
  ├─ Payments: 234
  ├─ Expenses: 108
  └─ Adjustments: 0
  
  Posted entries: 1,234
  Trial balance: BALANCED ✓
```

#### Phase 3: Tax Reconciliation (4 PM)

```
Navigation: Accounting → Tax → Monthly Reconciliation

Verify:
☐ VAT collected (from sales)
  Expected: Revenue × 15% ÷ 1.15
  
☐ NHIL collected (from sales)
  Expected: Revenue × 2.5% ÷ 1.025
  
☐ GETFUND collected (from sales)
  Expected: Revenue × 2.5% ÷ 1.025
  
☐ Tax payable GL accounts
  ├─ Tax Payable - VAT
  ├─ Tax Payable - NHIL
  └─ Tax Payable - GETFUND
  
☐ Outstanding tax payments
  └─ If due date passed, flag for payment
```

#### Phase 4: Financial Reports (5 PM)

```
Generate and review:

1. Income Statement (P&L)
   ├─ Revenue: Should be positive
   ├─ COGS: Should be ~40-50% of revenue
   ├─ Gross profit: Should be ~50-60% of revenue
   ├─ Expenses: Compare to budget
   └─ Net profit: Monitor trend
   
2. Balance Sheet
   ├─ Assets: Total increase monthly?
   ├─ Liabilities: Tax payable due?
   ├─ Equity: Retained earnings accumulated?
   └─ Balance: Assets = Liabilities + Equity?
   
3. Cash Flow
   ├─ Operating: Usually positive (cash sales)
   ├─ Investing: Equipment purchases
   ├─ Financing: Loans/owner draws
   └─ Ending cash: Should match bank balance
   
4. Trial Balance
   ├─ All accounts listed with balances
   ├─ Total debits = Total credits
   └─ All accounts used (no zero balances)
```

#### Phase 5: Director Review & Approval (5 PM - 6 PM)

```
Director reviews financial reports:
☐ Revenue trend acceptable?
☐ Profitability acceptable?
☐ Cash position healthy?
☐ No unusual account balances?
☐ Trial balance balanced?

Director approves close:
☐ Click "Approve Month-End Close"
└─ Status changes: OPEN → CLOSED

System prevents further changes:
├─ New sales not allowed for closed month
├─ Can only correct via journal entries
└─ Audit trail maintained
```

#### Phase 6: Post-Close Reconciliation (9 AM next day)

```
Verification by accountant:

☐ All transactions posted
  SELECT COUNT(*) FROM acc_postings 
  WHERE fiscal_period_id = [current month];
  
☐ Inventory matches GL
  SELECT SUM(quantity) FROM inv_stock_levels
  = (opening + purchases - COGS)
  
☐ Customer AR matches GL
  SELECT SUM(amount_due) FROM customers
  = AR account balance
  
☐ Tax payable matches GL
  SELECT SUM(tax_collected) FROM sales
  = Tax payable account balance
  
☐ All reconciliations documented
  └─ File in /reconciliations/[year]/[month].xlsx
```

---

## MONITORING & ALERTING

### Monitoring Infrastructure

```
Tool Stack:
├─ Metrics: Prometheus + Grafana
├─ Logs: ELK Stack (Elasticsearch, Logstash, Kibana)
├─ Uptime: StatusPage.io
├─ APM: New Relic / Datadog (optional)
└─ Synthetic Monitoring: Pingdom
```

### Key Metrics to Monitor

```
Application Level:
├─ Response time (target: < 2 sec, P95 < 3 sec)
├─ Error rate (target: < 0.5%)
├─ Request rate (monitor for traffic spikes)
├─ Active sessions (monitor for ddos)
└─ Queue job processing time (target: < 5 sec)

Database Level:
├─ Connection count (alert: > 100)
├─ CPU usage (alert: > 80%)
├─ Memory usage (alert: > 85%)
├─ Disk space (alert: < 20% free)
├─ Replication lag (alert: > 5 seconds)
└─ Slow queries (log: > 1 second)

Infrastructure Level:
├─ Disk I/O (alert: > 90% utilized)
├─ Network throughput (alert: > 80% capacity)
├─ CPU usage (alert: > 85% for 5+ minutes)
├─ Memory available (alert: < 512 MB free)
└─ Swap usage (alert: > 0% used)
```

### Alert Escalation

```
Alert Severity    Response Time    Escalation
─────────────────────────────────────────────
P1 Critical       Immediate         On-call Engineer
  (System down)                     + Manager + CEO

P2 High          15 minutes        On-call Engineer
  (Functional issue)               + Team Lead

P3 Medium        1 hour            Team Lead
  (Performance issue)              + Engineer

P4 Low           Next business day Operations
  (Information/warning)            Team
```

### Example Alert Rules (Prometheus)

```yaml
groups:
  - name: bisamapos_alerts
    rules:
      # P1: System Down
      - alert: SystemDown
        expr: up{job="bisamapos"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Bisama POS system down"
          action: "Check application server logs"

      # P2: High Error Rate
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: high
        annotations:
          summary: "Error rate > 5%"
          action: "Check error logs for root cause"

      # P3: Slow Response
      - alert: SlowResponse
        expr: histogram_quantile(0.95, http_request_duration_seconds) > 3
        for: 10m
        labels:
          severity: medium
        annotations:
          summary: "P95 response time > 3 seconds"
          action: "Analyze slow queries"

      # P4: Low Disk Space
      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.2
        for: 15m
        labels:
          severity: low
        annotations:
          summary: "Disk space < 20%"
          action: "Plan cleanup/expansion"
```

---

## INCIDENT RESPONSE

### Incident Response Plan

```
Incident Severity     Response Team        Target Resolution
────────────────────────────────────────────────────────────
P1 System Down        All hands on deck     30 minutes
  • No users can login
  • Sales cannot process
  • Database unavailable

P2 Partial Outage     Lead + 2 Engineers   2 hours
  • Some branches affected
  • Some features broken
  • Data integrity intact

P3 Degraded Service   1 Engineer + Lead     4 hours
  • Slow response times
  • Some errors
  • Workaround exists

P4 Minor Issue        Engineer             Next business day
  • Cosmetic problems
  • Information only
  • No impact to business
```

### P1 Critical Incident Response (System Down)

```
T+0 minutes (Incident Detected):
  1. Page on-call engineer (phone + SMS)
  2. Open war room (Zoom/Teams)
  3. Start incident timer
  4. Declare incident status (public notification)
  5. Post "investigating" on status page

T+5 minutes:
  1. Gather initial information:
     ├─ When did it start?
     ├─ What changed recently?
     ├─ Error messages in logs?
     └─ How many tenants affected?
     
  2. Check system status:
     ├─ Nginx running? (systemctl status nginx)
     ├─ PHP-FPM running? (systemctl status php-fpm)
     ├─ MySQL running? (systemctl status mysql)
     └─ Redis running? (systemctl status redis)
     
  3. Check logs:
     ├─ /var/log/nginx/error.log (last 50 lines)
     ├─ /var/log/php-fpm.log (last 50 lines)
     ├─ /var/log/mysql/error.log (last 50 lines)
     └─ Storage/logs/laravel.log (latest)

T+10 minutes (Triage):
  Most likely causes in order:
  
  1. Database connection issues
     ├─ Check: mysql -u root -p -e "SELECT 1;"
     ├─ Check replication lag (if applicable)
     └─ Fix: Restart MySQL if hung (last resort)
     
  2. Out of memory
     ├─ Check: free -h (available memory)
     ├─ Check: top (memory hogs)
     └─ Fix: Kill queue jobs if needed
     
  3. Disk full
     ├─ Check: df -h (free space)
     ├─ Check: du -sh /storage/* (big directories)
     └─ Fix: Delete old logs / archives
     
  4. Recent code deploy
     ├─ Check: git log --oneline -5 (recent commits)
     ├─ Check: PHP syntax errors (php -l app/*)
     └─ Fix: Rollback previous deploy
     
  5. Network connectivity
     ├─ Check: ping 8.8.8.8 (internet access)
     ├─ Check: telnet database.host 3306 (db access)
     └─ Fix: Notify ISP if needed

T+15-30 minutes (Mitigation):
  Start with quickest fixes:
  
  a) Restart services (90% fix rate)
     systemctl restart php-fpm
     systemctl restart redis
     systemctl restart nginx
     
     Wait 2 minutes, test access
     
  b) If still down, check database
     MySQL restart (last resort, may take 5+ min)
     systemctl stop mysql
     systemctl start mysql
     
     Monitor: tail -f /var/log/mysql/error.log
     
  c) If DB restart fails, restore from backup
     (See Backup & Restore section)
     
  d) If code issue, rollback
     git revert [last commit]
     php artisan migrate:rollback
     php artisan cache:clear
     systemctl restart php-fpm

T+30 minutes:
  If still not resolved:
  ├─ Escalate to hosting provider (if cloud-hosted)
  ├─ Get help from developers (if code issue)
  └─ Prepare customer communications

Throughout Incident:
  ✓ Update war room every 5 minutes
  ✓ Update status page every 10 minutes
  ✓ Document all actions taken
  ✓ Save all log files for post-mortem
  ✓ Communicate ETA to customers

After Resolution:
  1. Verify all systems:
     ├─ All tenants can login
     ├─ Test sales processing
     ├─ Test accounting posting
     └─ Monitor for 1 hour
     
  2. Communicate resolution
     ├─ Close status page incident
     ├─ Send all-clear email
     └─ Estimate impact (data loss, etc.)
     
  3. Post-mortem (within 24 hours)
     ├─ What happened?
     ├─ Why it happened?
     ├─ What we did to fix it?
     ├─ What we'll do to prevent it?
     └─ Root cause action items
```

---

## CHANGE MANAGEMENT

### Change Request Process

```
All changes follow this process:
1. Planning
2. Approval
3. Testing
4. Deployment
5. Verification
6. Documentation

Deployment Window:
├─ Sunday 2 AM - 6 AM (low traffic)
├─ 30-minute maximum downtime allowed
├─ Rollback plan required
└─ Customer notification 24 hours before
```

### Pre-Deployment Checklist

```
Code Changes:
☐ Code review approved (at least 2 engineers)
☐ All tests passing (unit + integration)
☐ No breaking changes to APIs
☐ Database migrations tested (can rollback)
☐ No hardcoded secrets
☐ Documentation updated

Infrastructure Changes:
☐ Capacity planning done
☐ Configuration tested on staging
☐ Rollback plan documented
☐ Monitoring updated for new metrics
☐ Team trained on new processes

Communication:
☐ Customer notification sent
☐ Support team briefed
☐ On-call engineer assigned
☐ War room setup ready
☐ Incident commander identified
```

### Deployment Procedure

```bash
#!/bin/bash
# /scripts/deploy.sh

ENVIRONMENT=$1  # staging or production
VERSION=$2      # git tag to deploy

if [ $ENVIRONMENT = "production" ]; then
  read -p "Deploy to PRODUCTION? Type 'yes': " CONFIRM
  [ "$CONFIRM" != "yes" ] && exit
fi

# Step 1: Update code
cd /var/www/bisamapos
git fetch origin
git checkout $VERSION

# Step 2: Run migrations
php artisan migrate --force

# Step 3: Clear cache
php artisan cache:clear
php artisan config:clear
php artisan route:clear

# Step 4: Rebuild indexes (Laravel)
php artisan optimize

# Step 5: Verify application health
php artisan tinker << EOF
  App::make('db')->connection()->getPdo();
  Cache::get('test') ? 'OK' : Cache::put('test', true, 1);
  exit('Health check passed');
EOF

# Step 6: Restart services
systemctl restart php-fpm
systemctl restart queue:work

# Step 7: Smoke test
curl -f https://bisamapos.com/api/health || {
  echo "Health check failed, rolling back!"
  git checkout main
  php artisan migrate:rollback --force
  systemctl restart php-fpm
  exit 1
}

echo "Deployment successful!"
```

---

## SECURITY OPERATIONS

### Access Control

```
SSH Access:
├─ Only key-based authentication (no passwords)
├─ 2FA required for server access
├─ Public key in ~/.ssh/authorized_keys
├─ Private keys stored in 1Password

Database Access:
├─ Read-only user for reporting
├─ Admin user for maintenance (limited)
├─ No direct database access from application
├─ Connection pooling enforced

Application Access:
├─ User roles: admin, director, manager, user
├─ Role-based permissions enforced in middleware
├─ API tokens stored in Laravel Sanctum
├─ Tokens expire after 1 year
```

### Security Patching

```
Linux/OS Patches:
├─ Check monthly: apt list --upgradable
├─ Apply patches: apt upgrade && apt autoremove
├─ Schedule: Sunday 2-4 AM
├─ Reboot if kernel updated

PHP/Laravel Updates:
├─ Check quarterly: composer outdated
├─ Review changelog for breaking changes
├─ Test in staging first
├─ Deploy to production per change management

Database Patches:
├─ Check quarterly: mysql --version
├─ Apply patches during maintenance window
├─ Test replication (if applicable)
├─ Verify backup/restore works after

Third-Party Libraries:
├─ Check monthly: composer audit
├─ Fix high-severity vulnerabilities immediately
├─ Fix medium within 1 week
├─ Low can wait until next release
```

### Audit Logging

```
What gets logged:
├─ All user logins (IP, timestamp, success/failure)
├─ All data exports (who, what, when)
├─ All financial postings (user, amount, GL account)
├─ All tax adjustments (reason, amount, approver)
├─ All access to archived data
├─ All admin actions (user creation, etc.)

Log retention:
├─ Application logs: 30 days
├─ Database transaction logs: 90 days
├─ Audit logs: 7 years (for compliance)
├─ Backup logs: 1 year

Log analysis:
├─ Daily: Check for authentication failures
├─ Weekly: Check for unusual activities
├─ Monthly: Full audit trail review
├─ Quarterly: Security incident analysis
```

---

## TROUBLESHOOTING GUIDE

### Problem: "Tenant Not Found"

**Symptom:**  
User visits domain but sees "Tenant not found" error

**Root Causes:**
1. Domain not registered in system
2. Domain configuration error in DNS
3. TenantMiddleware not extracting domain correctly
4. Tenant status set to inactive

**Resolution:**

```bash
# 1. Check tenant registration
mysql -u root -p << EOF
SELECT id, name, domain, status FROM tenants 
WHERE domain = 'acme.bisamapos.com';
EOF

# If not found:
INSERT INTO tenants (name, domain, status, created_at) 
VALUES ('ACME Company', 'acme.bisamapos.com', 'active', NOW());

# If found but inactive:
UPDATE tenants SET status = 'active' 
WHERE domain = 'acme.bisamapos.com';

# 2. Verify DNS
nslookup acme.bisamapos.com
# Should resolve to server IP

# 3. Check TenantMiddleware logs
tail -f storage/logs/laravel.log | grep -i "tenant"

# 4. Test middleware directly
php artisan tinker
>>> app()->instance('tenant', \App\Models\Tenant::where('domain', 'acme.bisamapos.com')->first());
>>> app('tenant')->id; // Should return tenant ID
```

---

### Problem: "Accounting Posting Failed"

**Symptom:**  
Sale created but posting_status remains 'pending' instead of 'posted'

**Root Causes:**
1. Chart of accounts incomplete
2. GL accounts not properly configured
3. Journal posting service error
4. Database transaction conflict

**Resolution:**

```bash
# 1. Check GL account configuration
mysql -u root -p << EOF
SELECT id, account_code, name, account_type, status
FROM acc_accounts
WHERE tenant_id = 15 AND status = 'active';
EOF

# Should see: Asset, Liability, Equity, Revenue, COGS, Expense accounts

# 2. Check unposted transactions
SELECT id, invoice_number, posting_status, error_message
FROM sales
WHERE posting_status = 'failed'
LIMIT 5;

# 3. Manually post unposted transactions
php artisan accounting:post-unposted --tenant=15

# 4. Check logs
tail -f storage/logs/laravel.log | grep -i "posting"

# 5. If persistent, check disk space
df -h /var/www/bisamapos/
# Must have > 1 GB free
```

---

### Problem: "High Memory Usage / Out of Memory"

**Symptom:**  
"Allowed memory size exceeded" error, application becomes unresponsive

**Root Causes:**
1. Queue jobs not completing
2. Large dataset export (PDF/Excel)
3. Memory leak in application
4. Too many concurrent connections

**Resolution:**

```bash
# 1. Check memory usage
free -h
# Should have > 1 GB available

# 2. Identify memory hogs
top -b -o +%MEM | head -10

# 3. If queue jobs stuck:
php artisan queue:failed      # See failed jobs
php artisan queue:retry all   # Retry failed jobs
# If still high:
redis-cli FLUSHDB            # Clear queue (last resort)

# 4. Increase PHP memory limit (temporary)
php -d memory_limit=512M artisan command:name

# 5. Permanent fix: Edit /etc/php/8.2/fpm/php.ini
memory_limit = 512M
max_execution_time = 300
systemctl restart php-fpm

# 6. Monitor for leaks
php -d memory_limit=512M artisan queue:work --tries=1 --timeout=300

# 7. If data export too large:
# Implement pagination / streaming export instead
```

---

**End of Operations Documentation**

*Contact: ops@bisamatech.com*  
*Escalation: operations-manager@bisamatech.com*  
*Last Updated: January 24, 2026*