Skip to content

Home

Jump to bottom
PranitSh edited this page Sep 22, 2023 · 2 revisions

Welcome to the keyflare wiki! I am still working on the formatting for the below, will be fixed soon!

keyflare package

                    • Submodules

keyflare.gui module

class keyflare.gui.GUI

Bases: "object"

This class named GUI (Graphical User Interface) serves as the primary interface for interacting with KeyFlare. It includes a coordinate selector, a keyboard input detector, color selector, etc. It continuously narrows down the location of your intended mouse click until it has been resolved, clicking automatically without a mouse.

Attributes: root (Tk object): The root Tk window for the KeyFlare application. input_char (str): The character input by the user. y (ImagePipeline object): An instance of the ImagePipeline class used for image processing and coordinate finding. exit_flag (bool): Flag to indicate whether to exit the application or not by the Usages class. color (tuple): The selected input method's color represented in (R, G, B) format. label: A tk Label to be shown in the application. temp_name: Name of the temporary file.

Methods: run(clicks: int): Runs KeyFlare's GUI process for selecting a coordinate. selecting_coordinate(clicks: int): Manages the process of selecting a coordinate on the keyboard image. on_key(event): Filters and updates available key options on a keyboard input. exit_app(): Gracefully exits the KeyFlare application. select_color(): Opens a color selection dialog and updates the selected color in the GUI. rgb_to_hex(rgb: tuple): Converts an RGB color tuple to its hexadecimal representation. focus_window(): Sets the application window to the foreground and ensures it has focus.

color = (248, 93, 94)

exit_app()

For exiting the KeyFlare application gracefully. It sets an exit flag and closes the application window.

Args: None

Returns: None

Example: >>> gui = GUI() >>> gui.exit_app()

exit_flag = False

focus_window()

Sets the application window to the foreground and ensures it has focus, no matter what. It is against the purpose of the application to find you have to use your mouse to click the image shown.

Args: None

Returns: None

Example: >>> gui = GUI() >>> gui.run() >>> gui.focus_window()

input_char = ''

label = None

on_key(event)

Filters and updates available key options on a keyboard input event. Narrows down the choices to find the spot to click.

Args: event (tkinter.Event): The keyboard event containing information about the pressed key.

Returns: None

Example: To use this method, you can define it as an event handler for a tkinter window like this: >>> self.root.bind("<Key>", self.on_key)

rgb_to_hex(rgb)

Converts an RGB color tuple to its hexadecimal representation.

Args: rgb (tuple): A tuple containing three integers representing the red, green, and blue color channels, respectively.

Returns: str: The hexadecimal representation of the RGB color.

Example: >>> gui = GUI() >>> color_tuple = (255, 0, 128) >>> hex_color = gui.rgb_to_hex(color_tuple) >>> print(hex_color) "#FF0080"

root = None

run(clicks)

Runs KeyFlare's graphical user interface process for selecting a coordinate.

Args: clicks (int, required): The number of mouse clicks to perform when a place to click has been found.

Returns: None

Notes:

  • Users can press a key on their physical keyboard, and the GUI will respond by narrowing down the available key options based on the input character.
  • If no matching key is found, the GUI will display color selection options and allow the user to exit the application.

Example: >>> gui = GUI() >>> gui.run(clicks=2)

select_color()

Opens a color selection dialog and updates the selected color in the GUI. The selected color is then displayed in the GUI, and the RGB values are updated.

Args: None

Returns: None

Example: >>> gui = GUI() >>> gui.run() >>> gui.select_color()

selecting_coordinate(clicks)

Manages the process of selecting a coordinate on the keyboard image, clicking the point at the end or showing a preference menu at the end if no point was found.

Args: clicks (int, required): The number of mouse clicks to perform when a place to click has been found.

Returns: None

Example: >>> gui = GUI() >>> gui.run(clicks=2) # Notice: please do not run this by itself

temp_name = None

y = <keyflare.image_pipeline.ImagePipeline object>

keyflare.image_pipeline module

class keyflare.image_pipeline.ImagePipeline

Bases: "object"

The *ImagePipeline* class performs a series of image processing tasks on a screenshot captured by the *System* class. It extracts coordinates of possibly clickable areas, removes overlapping bounding boxes, and labels the remaining boxes alphabetically. The processed data can then be used for various purposes, such as creating a graphical user interface (GUI) for navigating with your computer!

Attributes: original_image (None): A placeholder for the original screenshot image. coordinate_data (list): A list to store coordinate data, including bounding boxes of clickable areas. x (System): An instance of the *System* class used for capturing screenshots.

Methods:

  • run(): Executes the image processing pipeline.
  • processing_image(): Extracts contours from the original image into the coordinate_data.
  • processing_data(): Processes coordinate data to remove overlapping boxes and label the remaining boxes, recycling coordinate_data.

Notes:

  • The *ImagePipeline* class relies on the *System* class for capturing the original screenshot.
  • The processed results can be accessed using the
  • coordinate_data* attribute.

Example: >>> pipeline = ImagePipeline() >>> pipeline.run() # Executes the image processing pipeline >>> coordinate_data = pipeline.coordinate_data >>> print(len(coordinate_data)) # Output: Number of labeled bounding boxes

coordinate_data = []

old_collecting_data()

Extracts data from an image using optical character recognition.

Args: self (object): An instance of this method's class "Identifier."

Returns: None.

Raises: None.

Notes: The *self.processed_image* variable must be set to the input image. The extracted data is stored as a list of lists, where each inner list represents a region of interest following the format [left, top, right, bottom, confidence, text].

old_processing_data()

Filters a list of text data chunks based on their properties into coordinate data.

Args: self (object): An instance of this method's class "Identifier."

Returns: None.

Raises: None.

Notes: The *self.data* variable must be set to a list of data chunks before calling this method. Each data point should follow the format [left, top, width, height, confidence, and extracted text]. This method filters the data points based on their bounding boxes using the intersection over union (IoU) method. The *self.data* variable is updated to contain the filtered list. The *self.coordinate_data* variable is also updated to contain a dictionary that maps each chunk's left and top coordinates to a unique alphabet string identifier.

old_processing_image()

Performs image processing on an input .png image stored in a PIL Image.

Args: self (object): An instance of this method's class "Identifier."

Returns: None.

Raises: None.

Notes: The processed image is stored in the *self.processed_image* variable, and the original image is converted to RGB format and stored in the *self.original_image* variable. The

  • self.converted_image* variable stores a copy of the converted image as a numpy array.

old_run()

The *Identifier* class' initializer.

Args: x (object, optional): An instance of the *Main* class. Defaults to None.

Raises: None.

Notes: Initializes the *Identifier* class and its methods. This initializer automatically performs the usage pipeline.

original_image = None

processing_data()

Process the coordinate data to remove overlapping boxes and label the remaining boxes.

This method processes the coordinate data extracted from the image processing step. It includes the following steps: 1. Index the coordinate_data, which technically holds bounding boxes before this step, into the R-tree spatial indexing algorithm. 2. Remove overlapping bounding boxes. 3. Labeling the remaining bounding boxes alphabetically.

Args: None

Returns: None

Notes:

  • It uses the R-tree spatial index for efficient data manipulation.
  • This method is a part of the image processing pipeline.
  • The results are stored in the class's *coordinate_data* attribute.

Example: >>> pipeline = ImagePipeline() >>> pipeline.run() # Executes the image processing pipeline >>> coordinate_data = pipeline.coordinate_data >>> print(len(coordinate_data)) # Output: Number of labeled bounding boxes

Further Improvements:

  • Currently, if an item is overlapping another bounding box by 10 pixels on any side of the box, it is added to a list to be removed. An adaptive method designed to get a specific number of boxes would work better.

processing_image()

Processes the original image to extract contours into coordinate data. After converting an image from grey to black and white with adapative thresholding, the use of morpohological operations expands bright spots and contracts dark spots. The results, the contours of the image with and without morphological operations, are stored combined for further processing.

Args: None

Returns: None

Notes:

  • This method is a part of the image processing pipeline.
  • It uses OpenCV (cv2) functions for image manipulation.
  • The results are stored in the class's *coordinate_data*.

Example: >>> pipeline = ImagePipeline() >>> pipeline.run() # Executes the image processing pipeline >>> original_contours = pipeline.original_contours >>> processed_contours = pipeline.processed_contours >>> coordinate_data = pipeline.coordinate_data >>> print(len(coordinate_data)) # Output: Number of extracted bounding boxes

run()

Executes the image processing pipeline. This method starts the image processing pipeline after taking a screenshot. The results are stored in the class as an attribute, which is then displayed through the GUI class.

Args: None

Returns: None

Notes:

  • The method relies on the *System* class for capturing the original image.
  • The processed results can be accessed using the class's attributes.

Example: >>> pipeline = ImagePipeline() >>> pipeline.run() >>> coordinate_data = pipeline.coordinate_data >>> print(len(coordinate_data)) # Output: Number of clickable places found

x = <keyflare.system.System object>

keyflare.system module

class keyflare.system.System

Bases: "object"

This class provides methods for capturing screenshots and simulating mouse actions.

Attributes: None

Methods:

  • image(): Take a screenshot.
  • mouse(dataPoint, clicks): Move the mouse pointer to specified screen coordinates and perform the given number of mouse clicks.

Notes: This class only uses the PyAutoGUI library for system interactions. Please refer to the README for KeyFlare the latest installation instructions and for bypassing any potential OS limitations for full compatibility.

Example: >>> system = System() >>> screenshot = system.image() >>> print(screenshot.shape) # Output: (screen_height, screen_width, 3) >>> dataPoint = (500, 300) # Example screen coordinates >>> system.mouse(dataPoint) # Performs a single left-click at (500, 300)

image()

Take a screenshot without your mouse in the image.

Returns: np.ndarray: A NumPy array representing the screenshot image.

Notes: This method uses the *pyautogui.screenshot()* function to capture a screenshot of the entire current screen. PyAutoGUI has a number of system requirements that differ depending on the OS. Please check the README for the latest installation instructions for KeyFlare.

Example: >>> system = System() >>> screenshot = system.image() >>> print(screenshot.shape) # Output: (screen_height, screen_width, 3)

mouse(dataPoint, clicks)

Moves the mouse pointer to a specified screen coordinate and then performs a number of clicks.

Args: dataPoint (tuple or list, required): A tuple or list representing the target screen coordinates (x, y). clicks (int, required): The number of mouse clicks to perform.

Returns: None

Notes: This method uses the *pyautogui.moveTo()* and

  • pyautogui.click()* functions to perform these mouse actions. Again, PyAutoGUI has a number of system requirements that differ depending on the OS. Please check the README for the latest installation instructions for KeyFlare.

Example: >>> system = System() >>> dataPoint = (500, 300) # Example screen coordinates at 500 right and 300 down of the top-right corner of the screen. >>> system.mouse(dataPoint) # Performs a single left-click at (500, 300)

keyflare.usages module

class keyflare.usages.Usages

Bases: "object"

This class determines the way to use keyflare if called through main, whether it is __main__, main(), or maybe something else in the future.

Attributes: args (list): Command-line arguments passed to the program. platf (str): The platform (e.g., 'Windows', 'Linux', 'Darwin') on which the program is running. z (GUI): An instance of the GUI class. clicks (int, defaults to 1): The number of clicks to perform when a keyboard shortcut is triggered.

Methods: __init__(): Initializes the Usages class and sets up necessary attributes. runType(): Calls the shortcut method to listen for keyboard shortcuts. shortcut(): Listens for specific keyboard shortcuts and triggers GUI actions accordingly. programmatic(): Executes GUI actions programmatically based on command-line arguments.

Notes:

  • This class depends on the external library 'pynput' and standard library 'platform'.
  • It is designed to work with a GUI application represented by the 'GUI' class.
  • The keyboard shortcuts are defined in the 'start_combination' list. The chosen shortcut is alt+A.

Examples: >>> usage = Usages() >>> usage.runType()

args = None

clicks = 1

platf = None

programmatic()

Executes GUI actions programmatically based on command-line arguments.

Args: None

Returns: None

Notes:

  • This method is typically called from the command line with the desired number of clicks as an argument.

Examples: "`sh python script.py 3 `"

runType()

Currently unimplemented. Always runs *self.shortcut()*. In the future, this will call the appropriate method.

Args: None

Returns: None

shortcut()

Listens for keyboard shortcuts and triggers the GUI action if the shortcut is pressed.

Args: None

Returns: None

Notes:

  • This method relies on *pynput*, which has platform specific limitations. Please read the README for bypassing them.

z = None

keyflare.usages.main()

The main entry point of the script.

Args: None

Returns: None

tests.test_image_pipeline module

keyflare.tests.test_image_pipeline.test_image()

Fixture that loads a test image from a file for consistent testing.

Returns: np.ndarray: The test image as a NumPy array.

keyflare.tests.test_image_pipeline.test_image_pipeline(benchmark, test_image)

Benchmark test for improved ImagePipeline's run method. It measures the time take to process the image and extract coordinates of clickable locations.

Assertions:

  • Ensures that coordinate_data is a list.
  • Ensures that coordinate_data contains data.
  • Ensures that each item in coordinate_data is a pair of (key, coordinates).

keyflare.tests.test_image_pipeline.test_old_image_pipeline(benchmark, test_image)

Benchmark test for my older ImagePipeline process that used tesseract.

Assertions:

  • Ensures that coordinate_data is a dict. I previously used a dicts
  • Ensures that coordinate_data contains at least one coordinate.
  • Ensures that each item in coordinate_data is a pair of (key, coordinates).

keyflare.tests.test_system module

keyflare.tests.test_system.system_instance()

Fixture; creates an instance of the System class.

Returns: System: An instance of the System class.

keyflare.tests.test_system.test_capture_screenshot_benchmark(system_instance, benchmark)

Benchmark test for taking a screenshot and converting it a numpy.ndarray as part of System's image() method.

Args: system_instance (System): An instance of the System class. benchmark: The benchmark fixture provided by pytest-benchmark.

Assertions:

  • Ensures that the captured screenshot is an instance of np.ndarray.

Clone this wiki locally