Context & Problem Space
The business context
Diligent's governance platform migrates clients from legacy systems — including BoardDocs — into a unified board management solution. Before any migration kicks off, analysts must audit the client's existing BoardDocs data to identify what migrates cleanly, what needs transformation, and what will break.
The bottleneck
This pre-migration audit was entirely manual. An analyst would log into BoardDocs, navigate through committees, meetings, video links, templates, policies, and library documents — clicking through hundreds of pages, copying counts into spreadsheets, and manually flagging issues like duplicate meetings or non-migratable video formats.
A single audit took ~2 full working days per client. The analyst had to check 8 distinct data areas, cross-reference formats, detect duplicates by comparing titles and dates, and then compile a client-facing report in plain English — all before migration could even begin.
Target users
Primary: Migration analysts and Customer Success Specialists on the Bengaluru team who perform pre-migration audits before every client onboarding.
Secondary: Client stakeholders who receive the audit report and need to resolve data issues before migration proceeds.
Baseline friction
- ~2 days of manual clicking per client audit
- Human error in duplicate detection (same title + same date = duplicate)
- No standardised report format — each analyst produced different outputs
- Video link platform identification done by visual inspection (YouTube, Swagit, Vimeo, or non-migratable)
- Client-facing report mixed internal technical language with action items
Discovery & Technical Constraints
Technical landscape
BoardDocs is a legacy governance platform with no public API. All data — committees, meetings, video embeds, policy attachments, templates, library documents — is rendered as DOM content in the browser. Extracting this data required browser automation, not API calls.
Architecture decision: Selenium + BeautifulSoup on Flask
The first week (Mar 9–15) was a dedicated trial week where the full team evaluated tool and technology options before any building started. The decision was locked by end of Mar 15: Python with Selenium for browser automation, BeautifulSoup for DOM parsing, and Flask as the local dashboard server.
Key technical constraints
- DOM fragility: ~5% of audit failures trace back to DOM structure changes on the BoardDocs side — elements move, class names change, and pages re-render unpredictably
- Scale: Tool must handle 500+ meetings and 50+ committees per client without session timeouts or memory exhaustion
- Concurrency: Sequential scraping was too slow — needed parallel workers for committee-level audits
- Video platform detection: Must distinguish between YouTube, Swagit, Vimeo, and non-migratable formats by parsing embedded URLs and identifying platform-specific patterns
- Duplicate detection: Meetings with identical titles AND identical dates flagged as duplicates — requires cross-referencing across the full meeting set
- Client report isolation: The report must contain zero technical language — no mention of Selenium, crawlers, or scripts — only plain-English action items
Stakeholder trade-offs
The team included 8 people across development, UI design, and project management. Area ownership was distributed — each person built and QA'd their own area with no separate QA role. This reduced handoff overhead but required tight coordination during integration.
The decision to run a full trial week before Sprint 1 meant the 8-week timeline effectively became 7 weeks of build time. This was a deliberate trade-off: one week spent evaluating options prevented mid-sprint pivots on tooling.
Solution & Product Ownership
PRD scope and structure
Authored the complete PRD (v3) covering all 8 audit areas with explicit acceptance criteria, dashboard specification, client report specification, sprint plan, and go-live approval checklist. Every area had defined ownership, QA responsibility, and measurable output expectations.
The 8 audit areas
Architecture: 3 concurrent workers
The platform runs 3 concurrent Selenium workers — each handling a separate committee's full audit in parallel. This was the key throughput multiplier: instead of sequentially crawling 50+ committees, the tool processes 3 at a time, reducing total scrape time from what would have been 12+ hours to approximately 6 hours.
Client report design decisions
The client report was designed as an action document, not a data summary. Page 1 opens with a plain-English bullet list of exactly what the client needs to fix. Below that, each issue gets a detailed breakdown so the client knows which specific records are affected.
What the report deliberately excludes: count summaries (those live on the internal dashboard only), technical language (no mention of Selenium, BeautifulSoup, crawlers), and internal team notes.
Execution & Cross-Functional Leadership
Sprint structure and PM cadence
The project ran on a structured 8-week timeline: 1 trial week + 7 weeks across 3 sprints. As PM, I ran a weekly checkpoint cadence — Monday goal confirmation, Wednesday mid-sprint review, Friday risk assessment — alongside daily standups.
Week-by-week execution
Trial week (Mar 9–15): Full team evaluated technology options. Decision locked on Selenium + Flask by end of week. No code written until the decision was final — a deliberate choice to prevent mid-sprint tooling pivots.
Weeks 1–2 (Mar 16–29): All 8 area owners started building simultaneously. Confirmed BD access for every team member. Briefed UI designers on all screen requirements so frontend and backend work ran in parallel. Reviewed early proof-of-concept outputs from each area.
Weeks 3–4 (Mar 30–Apr 12): Each person demoed their area progress. Validated outputs against acceptance criteria — specifically confirming duplicate meeting detection and video link platform identification were working correctly. Wireframes approved for all dashboard screens.
Week 5 (Apr 13–19): Integration sprint — all individual areas merged into the unified platform. Confirmed account name scrape, dashboard rendering, and client report generation were working end-to-end.
Weeks 6–7 (Apr 20–30): Full end-to-end review, UAT across all 8 areas, go-live checklist clearance. Zero critical bugs at UAT sign-off.
Cross-functional coordination
Managed alignment across 8 people — developers building individual audit areas, UI designers handling wireframes and frontend, and a project manager running daily execution. Each person owned both build and QA for their area — no handoff to a separate QA team. This reduced communication overhead but required tight acceptance criteria upfront (which the PRD provided).
Go-live approval process
Go-live required co-approval from myself (Product Manager) and the Team Lead. UAT was conducted and signed off by the Project Manager before approval was sought. The go-live checklist covered every audit area with specific pass criteria — from correct Public/Private committee splits to zero technical language in the client report.
Outcomes & Key Takeaways
Verified business impact
All metrics below were presented to and verified by Executive Leadership.
Operational efficiency
Near-elimination of human error in pre-migration validation. Duplicate detection, video platform identification, and format scanning — all previously done by visual inspection — now run programmatically with consistent results across every audit.
Standardised client reporting. Every client now receives the same structured, plain-English report format. No more analyst-dependent variations in output quality or completeness.
Analyst bandwidth reclaimed. The hours freed up per audit are now redirectable to actual migration execution — the work that moves clients onto the platform, not the validation overhead that precedes it.
Key takeaways
1. The trial week paid for itself. Spending a full week evaluating tooling options before any code was written prevented the most common failure mode in internal tooling projects: mid-sprint technology pivots. The decision to use Selenium + Flask was locked once and never revisited.
2. DOM scraping is inherently fragile — design for it. ~5% of failures trace to BoardDocs DOM changes. This is a known and accepted constraint, not a bug. The architecture accounts for it with retry logic and error reporting that tells you exactly which element failed and why.
3. Scope discipline matters more than feature count. The PRD deliberately excluded features the team suggested — category breakdowns for templates, compliance scoring for policies, active/archived splits. Every cut kept the tool focused on what migration analysts actually need before kickoff: what exists, what's broken, and what the client needs to fix.
4. Shipping earns scope. This tool started as a personal automation to reduce my own audit workload. After demonstrating working output, leadership funded a team of 8 and a structured sprint plan. The initiative was earned through delivered outcomes, not a pitch deck.