Untitled47.ipynb - Colab

Untitled47.ipynb_ star File Edit View Insert Runtime Tools Help comment settings people Share format_list_bulleted search vpn_key folder code terminal Code Text Gemini expand_less chunk document get entities from document loop through chunks with prompt and make inverted index of entities->chunk indices take each entity’s indexes and make a summary tree, add nodes to chunks vectorise chunks from typing import List, Tuple, Iterator, Optional from dataclasses import dataclass import re from itertools import islice from dataclasses import dataclass, field from typing import List, Optional @dataclass(frozen=True) class Chunk:     text: str     token_count: int     start_idx: Optional[int] = field(default=None)     end_idx: Optional[int] = field(default=None)     embedding: Optional[List[float]] = field(default=None) def count_tokens(text: str) -> int:     """     Approximate token count using simple whitespace splitting.     Replace with your preferred tokenizer (e.g., tiktoken, GPT2Tokeni

saved by

• Sarvasv Kulpati