Datasources

Creating a DataSource

To add a data source in Dash Agent, you need to create a class that extends the DataSource abstract class. The DataSource class provides three getters: projectObjects, fileObjects, and webObjects, which allow you to specify the data sources for your agent.

1. WebDataObject

class SampleDataSource extends DataSource {
  @override
  List<WebDataObject> get webObjects => [
    // single page
    WebDataObject.fromWebPage('https://sampleurl.com'),
    // entire site
    WebDataObject.fromSiteMap('https://sample-web/sitemap.xml'),
  ];
}

Use this to index your web documentation, blogs (open and scrappable), and even github repositories and issues.

2. FileDataObject

class SampleDataSource extends DataSource {
  @override
  List<FileDataObject> get fileObjects => [
    FileDataObject.fromDirectory(
      Directory('directory_path_to_data_source')),
    FileDataObject.fromFile(File('your_file_path')),
  ];
}

Use this to index your local codebases, private content and specialized content created to assist agents.

Note: Only text based files like text, markdown, code etc is supported. PDFs or images and videos will not be indexed.

3. ProjectDataObject

class SampleDataSource extends DataSource {
  @override
  List<ProjectDataObject> get projectObjects => [
    ProjectDataObject.fromText('Data in the form of raw text'),
    ProjectDataObject.fromJson({
      'Sample API 1': 'description and usage',
      'Sample API 2': 'description and usage',
    }),
  ];
}

Provide locally created content from within the agent project.

Once you have defined your data sources using the appropriate getters, you can use them in your process steps to find relevant documents.

MatchDocumentStep

To find most relevant documents (top 3-5) from the entire data sources, you can use the MatchDocumentStep in your process steps.

Example usage:

final SampleDataSource sampleDataSource;
final userQuery = StringInput('Your query', optional: false);
final codeAttachment = CodeInput('Primary method');
 
final matchingDocuments = MatchDocumentOutput();
return [
  MatchDocumentStep(
    query: '$userQuery$codeAttachment',
    dataSources: [sampleDataSource],
    output: matchingDocuments,
  ),
  // Next: use the matchingDocuments output in the PromptQuery or other steps
];

• query: Query built by string interpolating user inputs or previous step outputs.
• dataSoures: List of data sources to search across.
• output: To be used in the next steps like PromptQuery or for other processing.

By leveraging data sources and the MatchDocumentStep, you can easily find relevant documents from your entire data sources and use them in your Dash Agent's processing pipeline.